Blue Flower

Domain-Specific Languages (DSLs) are a new buzzword these days. But DSLs have been around for a long time, and are not always computer related. For example, various notations for chess moves, like N-QB3 or Nf6, have been in use since at least the 1700s, and probably earlier. They can describe a chess game much more concisely than any natural language. Some SWAT teams use a visual DSL called SWATCOM, a series of hand signals that can say things like, “I see two men and one woman with guns, moving towards you.” These languages are very concise, but at a cost of being very narrow in scope. For example, neither chess notation nor SWATCOM can say “How did you like the new Britney Spears video?”

This same characteristic applies to computer DSLs. A programmer can use a standard programming language, such as C++, Java, or assembly language, to write a program. But sometimes you need to have programs written by subject-matter experts who are not programmers. Teaching them a general-purpose language is a major undertaking. But you can probably teach them a simple language tailored to their domain.

The RTSTRAN story

In the mid-1970s, I was working for a division of General Motors that needed a way to describe vehicle wiring diagrams to an automated test system. If you have ever looked at the wiring diagram in the back of the owner’s manual for your car, it is drawn like a plate of spaghetti, with devices located based on where they are in the car, and lines crisscrossing each other to connect the devices. It would be very difficult to enter this kind of diagram into the computer without making mistakes. Clearly, we needed another approach.

The first step was to get the Electrical department to redraw the electrical diagrams in the style of industrial ladder diagrams. These diagrams have vertical lines representing the power supply and ground, and a series of horizontal lines that go from the power supply wire to the ground, showing all the devices in between. There are as many lines as are required to describe all the circuits. No attempt is made to represent where the device is physically located.

Here is an example of how a horn circuit looked in the new style of diagram:

Once we had the wiring diagrams in this new form, we developed a domain-specific language, RTSTRAN, that could describe them. Here is how the diagram above could be described in this language:

Since the wiring diagrams were drawn in a standard way, it was fairly easy to teach someone how to convert them to RTSTRAN and enter them into the test system.

By the way, if I were doing this today, three decades later, I would consider having the engineers enter the diagrams using a graphical ladder diagram editor, which would eliminate re-keying the information completely. Here is one example of such an editor.

There are several other available ladder diagram editors, both proprietary and open source, as well as compilers that will compile the diagrams into XML. For more information about them, do a search on the Internet for “IEC 61131-3”, “LD”, and “compiler”. (IEC 61131-3 is an international standard that defines programming languages for programmable logic controllers. The LD section describes graphical ladder diagram representations.)

Once we had the diagrams entered in RTSTRAN, we needed to process them in order to convert them to tables that would be downloaded to the minicomputer that ran the test system. W.H. McKeeman had published a book called A Compiler Generator. It described, and provided the source code for, a compiler for a PL/I-like language called XPL, as well as a grammar analyzer that made parsing tables that could be used to process other languages. These were written in XPL. (It was a self-compiling compiler).

Luckily, Mary Pickett, Heinrik Schultz, and Fred Krull at General Motors Research Labs were working on various industrial computer languages and had ported the XPL compiler and analyzer to PL/I, so they could run on the IBM 370.

Using these programs, along with the RTSTRAN grammar (which was written in BNF), produced a language translator with stubs for processing each element of the language. All I had to do was fill in the processing in each stub, which was a lot easier than writing a compiler from scratch would have been.

Here is what the beginning of the RTSTRAN grammar specification looked like. (In all, there were 111 rules):

Today, there are several open-source compiler-generating tools such available, such as the following:

Other uses

You might be thinking “That’s nice, but I don’t have any electrical wiring diagrams to process in my product.” That may be, but before you reject the idea of a DSL, think about whether you have any complicated configuration information that the customer has to enter. For example, does the customer need to describe his network to your product, or perhaps the devices in his or her data center? For complicated input of this nature, a DSL can make it much easier and faster for the customer to enter the necessary information.

ColmoneBOL

Implementing a DSL does not have to be as complicated as writing a full compiler. For example, in the late 1980s, my colleague, Ron Colmone, worked on a security product for a minicomputer. He developed a DSL that, with a nod to COBOL, was nicknamed ColmoneBOL. The purpose of the DSL was to process command packets sent up from the minicomputer and issue the appropriate commands on the mainframe security system. There were many types of command packets, and the DSL provided the building blocks necessary to add conditional logic to some very powerful runtime functions. The DSL also provided a debugging/trace facility that assisted in diagnostics and debugging.

In this case, the DSL was processed by the MVS High-Level Assembler, using a macro that did an AREAD for the entire script, and then generated blocks that could be executed by the interpreter.

Domain-Specific Languages by Martin Fowler with Rebecca Parsons

This brings us to the main subject of this review, Domain-Specific Languages, by Martin Fowler with Rebecca Parsons. It was published in 2011 as part of the Martin Fowler Signature Series. The book begins with some introductory material to ease the reader into the topic of DSLs, then provides a number of chapters about various aspects of DSLs.

Chapter 1 begins with a hypothetical example that is simple and fun, but also lays out some of the reasons one might want to use a DSL. It describes a “Gothic Security System”, based on old Gothic movies, where one opens a secret panel in the wall by pulling the candle holder at the top of the stairs and tapping the wall twice, or something like that. It discusses how one might design a language so this system could be easily adapted for each haunted mansion where it is installed.

In the example, a customer wants a secret panel to open when she closes her bedroom door, opens a drawer in her dresser, and turns on the bedside light. The toy DSL for this project defines the events, shown below in green (the designations like D1CL refer to the inputs from the various sensors), the commands, shown below in red (the designations like PNUL refer to outputs to control locks and such), and then a series of states, shown below in blue. Each state lists events that cause a change of state, and what the new state is. They can optionally specify commands to be issued while in that state. The “resetEvents” section lists events that immediately put the system back into the idle state.

events
      doorClosed        D1CL
      drawerOpened      D2OP
      lightOn           L1ON
      doorOpened        D1OP
      panelClosed       PNCL
end

resetEvents
      doorOpened
end

commands
      unlockPanel       PNUL
      lockPanel         PNLK
      lockDoor          D1LK
      unlockDoor        D1UL
end

state idle
      actions {unlockDoor lockPanel}
      doorClosed => active
end

state active
      drawerOpened => waitingForLight
      lightOn => waitingForDrawer
end  

state waitingForLight
      lightOn => unlockedPanel
end

state waitingForDrawer
      drawerOpened => unlockedPanel
end

state unlockedPanel
      actions {unlockPanel lockDoor}
      panelClosed => idle
end

Here is a diagram that shows the state transitions of the finite state machine described by the above language. (The commands issued in each state are not shown.) The reset events are not shown, as they would make the diagram difficult to read. They would appear as lines from each node below the “idle” node, going back to the “idle” node and labeled with “doorOpened”.

The above language, while perhaps not intuitive, is concise, but flexible, since it can trigger actions based any series of events (e.g., close the door, open each dresser door in order, tap on the wall three times, turn on the TV, close each dresser door in order, etc.). And it doesn’t have a lot of syntactic noise, like semicolons and continuation characters.

The book mentions the possibility of using XML to configure the Gothic Security system, and quickly rejects it. XML has a lot of syntactic noise, and a lot of opportunities for syntax errors, with its nested tags, and opening and closing angle brackets that have to match, and that sort of thing. This all makes it fairly unfriendly to humans, particularly those who are not programmers by trade.

The book then talks about internal versus external DSLs. An external DSL is a standalone language, like the one described above. An internal DSL (sometimes referred to as an embedded DSL) is a general-purpose language, like Java or Ruby, that is warped, through special usage, into a DSL. For example, here is the same description of a Gothic Security system, but described in specially formatted Ruby.

event :doorClosed,    "D1CL"
event :drawerOpened,  "D2OP"
event :lightOn,       "L1ON"
event :doorOpened,    "D1OP"
event :panelClosed,   "PNCL"

command :unlockPanel, "PNUL"
command :lockPanel,   "PNLK"
command :lockDoor,    "D1LK"
command :unlockDoor,  "D1UL"

resetEvents :doorOpened

state :idle do
  actions :unlockDoor, :lockPanel
  transitions :doorClosed => :active
end

state :active do
  transitions :drawerOpened => :waitingForLight, 
              :lightOn => :waitingForDrawer
end 

state :waitingForLight do 
  transitions :lightOn => :unlockedPanel 
end 

state :waitingForDrawer do 
  transitions :drawerOpened => :unlockedPanel 
end 

state :unlockedPanel do 
  actions :unlockPanel, :lockDoor 
  transitions :panelClosed => :idle
end

This looks very much like the original DSL, with a bit more syntactic noise, but it is actually valid Ruby. The keywords event, command, resetEvents, and state are actually methods. The blocks like “state :unlockedPanel do” use a Ruby construct where the contents of the block, method calls and all, are passed to the method.

Thus it is possible to tack the user’s configuration specification onto the end of the program that defines the methods, and run the whole thing through the Ruby interpreter. This has pros and cons: you don’t have to write a separate compiler for your DSL, but it is syntactically noisier, and the interpreter will probably give confusing messages if the user makes any errors.

The rest of Chapter 1 deals with semantic models, code generation, language workbenches, and visualization. This is followed by 56 chapters in six parts:

  • Narratives
  • Common Topics
  • External DSL Topics
  • Internal DSL Topics
  • Alternative Computational Models
  • Code Generation

With 57 chapters total, the book seems a bit intimidating at first, and it would be, except for the way it is organized, which is similar to the “design patterns” books in the Martin Fowler Signature Series. The book consists of very short chapters, often just three or four pages, that talk about a particular topic. When that topic is referenced, the page where its chapter starts is given in parentheses, like this:

With each event declaration, I can create an event from the Semantic Model and put it into a Symbol Table (165).

Using the page number in the reference, rather than a chapter or section number, or referring to a footnote, means you do not have to go to the table of contents or look somewhere else to find the page you want. And the book has a built-in ribbon to hold your place, so if you want to spend some time in the referenced chapter, you do not have keep your finger on the page you came from to hold your position. This seems like a simple thing, but in practice, it makes it much easier to follow the references.

The short chapters, each about a single topic, make it easy to look up what you need to, and skip things you already know or do not need. If you are implementing a DSL on a deadline, rather than taking a college course on compilers, this organization works much better than the older compiler books that would have a large chapter on each general area of compiler design, long on theory and short on practice. This is a book for the working programmer who needs to implement a DSL.

Now, you may never need to implement a DSL. But it is also possible that doing so would make your life a lot easier, and you have just not considered the possibility. This book will help you decide, and show you some of the possibilities, without bogging you down with too much theory.

One parting thought: some of us still spend most of our working lives coding in assembly language. It is possible to implement a DSL in assembly language. (In fact, ColmoneBOL, mentioned earlier, was implemented in assembly language.) But there are so many more tools and facilities available in high-level languages, not to mention faster coding (studies have shown that programmers code about the same number of lines of code per day, regardless of whether they are coding in assembly language or a high-level languages, and high-level languages do more per line) and fewer error possibilities, that the cost of learning a new language may be outweighed by the improved efficiency it brings.

Software development looks like manufacturing if the unit of inventory is the unvalidated decision. —Alistair Cockburn

What do you do when you finish your work early in an iteration, and have a few days left in the sprint? Do you just pull in another story from the backlog? After all, there's plenty of work to do, why not get a jump on it?

No, that's not what you do. First, you remember that you are part of a team, so you see if anyone on the team needs your help to finish their work, which will help the team deliver on its commitments for the iteration.

OK, so you did that. Now can you pull in another story from the backlog? No!

In his post, 10 Things You Could Do When You’re Done Writing Code, John Krewson of VersionOne talks about the perils of pulling in more work from the backlog mid-iteration:

When you add additional work into the iteration, you’re increasing the batch size. This causes additional complication due to increased changes in the code base. It usually results in carryover, which has a demoralizing effect on the team. And it eliminates an opportunity to get feedback on the emergent product before additional work is embarked on.

Remember, the iteration was planned so that all members of the team can function at optimum efficiency, not so that the developers can program at optimum efficiency.

A lot of agile development ideas come from lean manufacturing, so it is informative to compare the two. In A Study of the Toyota Production System: From an Industrial Engineering Viewpoint, industrial engineer Shigeo Shingo says:

Workers can be kept busy with the following types of tasks during low-demand periods:

  • Repairing small leaks throughout the plant that have been neglected. (This saved one company close to $5,000 in water charges.)
  • Maintaining and repairing machines that have been negleced during the regular production cycle.
  • Practicing tool and die changes.
  • Fabricating jigs and fixtures for planned improvements.

Compare that with this list of things agile development team members should do, again from John Krewson's post:

  1. Refactor something you recently wrote.
  2. Add unit tests to an uncovered area of code.
  3. Find a technical tutorial on the Internet.
  4. Offer to pair with a developer on another team.
  5. Estimate open items in the backlog.
  6. Create some automated functional tests.
  7. Clean your desk.
  8. Ask your product owner about the next release.
  9. Help your product owner create acceptance criteria for items in the backlog.
  10. Implement a prioritized improvement suggestion from the last retrospective.

In both cases, the key thing is that you should not be doing work that you do not know you need. In manufacturing, you should not be making more parts than you need right now, because:

  • You don't know when you will need them.
  • If you make up a whole bunch of them in advance, there may be a design change before you need them, rendering them useless.
  • They take up space that could be used for manufacturing.

Similarly, when you bring in new stories from the backlog, you do not know if you are going to need them. When you do the end-of-sprint review, the customer feedback may send you off in another direction, and what you have coded will be wasted.

A parallel to what John Krewson said about improving the overall team efficiency, not just that of the developers, is the way manufacturing used to be done. Back in the day, the goal was to get 100% utilization out of the machines, since they were expensive. So, even if parts were not needed, they were manufactured and warehoused for later use, just to increase the machine utilization. And they were produced in large quantities, to reduce the effect that setup time had on utilization. And they were scrapped out when a design change occurred.

This is like what we used to do in software development, producing a lot of infrastructure programs because we thought we would need them someday. I have done this many times over the course of my career, and invariably, when I actually do need the code, I have to rework it. This is the genesis of the Extreme Programming rallying cry You Aren't Gonna Need It (YAGNI).

There are plenty of things to do with leftover time in an iteration that are better than producing new code that my may have to discard or rework.

I would like to thank John Krewson and Andrea Keeble of VersionOne, John for his excellent post that expressed some of these ideas more eloquently than I could, and Andrea for permission to quote much of the post here.

In the spirit of both disclosure and self-promotion, I would like to mention that my company, CA Technologies, has a joint venture with VersionOne to provide seamless integration with our own CA PPM (Project and Portfolio Management) product: http://www.versionone.com/ca/

When I attended General Motors Institute (now Kettering University) in the early 1970s, it didn't have the free-wheeling feel of many college campuses. It was part of General Motors back then, and shared a lot of the traditional General Motors conservatism.

One thing that really irritated me at the time was the class bells. It made me feel like I was still in High School, especially when my friends at other colleges told me that they were just expected to show up to class on time, without a bell prodding them.

But with a few more decades under my belt, I'm beginning to think the bells were not such a bad idea. Today we have lots of meetings, and most of them start and end on the hour. A few enlightened teams start their meetings at five after the hour, and end at five before hour, but not very many. And meetings frequently run over their time box, because the participants figure since it is only 10 or 15 minutes, it is easier to keep going than to schedule another meeting. But those 10 or 15 minutes don't belong to them. If participants are due in another meeting, all the people in the subsequent meeting need to wait for them.

That is where the class bells would help. If we had a bell (or better, a soft tone over the public address system) at five minutes after the hour and five minutes before the hour (perhaps the half-hour, too), it would remind people that they have to end their meeting so that people can get to their next one.

Actually, it would not be all that difficult to tie the room reservation system in with one of the desktop alerting systems, like SnapComms, NetSupport Notify, or Alertus, so when the meeting has exceeded its timebox, a message scrolls across projector screen saying "This meeting has ended. Please clear the room for the next meeting." This would avoid disturbing people who were not in the meeting.

Some would argue that trying to solve a behavioral problem with technology is focusing on the wrong thing. However, my experience has been that it is sometimes the only way. If you tell people to end their meetings on time, many will feign agreement, but continue to overrun their allotted timeboxes. If you use technology to make it inconvenient for them to exceed their timebox, they are then in the position of either putting up with the inconvenience, starting to behave right, or complaining about the end-of-meeting messages and explaining why their meeting is so important that it should be allowed to impinge upon other people's meetings.

It all depends on the company culture. In some companies, all you need to do is to get mid-level management to proclaim that meetings should not overrun their timeboxes, and set a good example by ending their own meetings when the timebox expires. In other companies, particularly mature companies where people are set in their ways, you may need to use technology to influence them.

Automated testing is a key part of agile development. If you are changing software in frequent small increments, you need to be sure that your new changes have not broken the existing code. Because the increments are usually between one and four weeks long, there is no time for test regimens that take weeks to complete. And manual tests become especially tedious (which leads to errors and oversights) if they have to be repeated frequently.

GUIs are frequently considered something difficult to test in an automated fashion. One thing that can help a lot is separating the production of data from the presentation of data, using the Model-View-Controller, Model-View-Presenter, or Model-View-ViewModel design patterns. (Here is a good overview of the differences between MVC, MVP, and MVVM.) In all three of these design patterns, the Model is the part of the system that contains its state and performs all calculations.

You can write automated unit test programs that call methods in the Model and check for expected results, just as with any module, without worrying about how it is presented on the screen. Testing the GUI itself is just a matter of making sure that when you click on a button, it sends the right signal to the ViewModel, and when you pass the View Model a particular value, it is properly displayed. Even if you have to do these tests manually, it is easy to write a test driver that tells you when each button has been clicked, and populates each display with dummy data. This is a lot easier than trying to figure out what you have to do to the system to make it display a particular thing.

There are some other benefits to separating the production and presentation of data. It can make it a lot easier to change the GUI without fear of messing up the data production. For example, Boonex Dolphin, a content-management system for social websites, does not directly generate its HTML web pages. Instead, it generates XML, which is run through an XML stylesheet (XSLT) to format it into a web page. This lets the web page designer concentrate on the design, without having to worry about the business logic. It also makes it easy to have several themes available, completely changing the look of the website, by just changing the XML stylesheets, without changing the business logic of the website.

Another example is XAML (Extensible Application Markup Language), an XML-based markup language for designing GUIs. It is used by Microsoft's Windows Presentation Framework and Silverlight application-development frameworks, as well as NoesisGUI, a platform-independent GUI middleware package that is available for Windows, OSX, and Linux, as well as several popular game and rendering engines.

A designer can design a GUI either visually, using Microsoft Blend or Microsoft Visual Studio, or by hand with a smart XAML editor like Kaxml that immediately displays how the XAML will be rendered. The designer can concentrate on what the page looks like, without having to get involved with how the data is calculated.

This allows separation of duties: someone who is good at design can design the GUI, and someone who is good at code and business logic can code the Model and ViewModel. Even if you are a one-person indie shop, and have to wear several hats, it is easier to be in a design headspace or a coding headspace, and not be constantly switching back and forth between the two.

Yet another benefit is code reuse. If your business logic is not tied to the GUI, it makes it a lot easier to reuse it in a different application.

Designing your GUI for automated testing helps you meet your sprint commitments, makes changing it easier, and makes your code more reusable.

What's not to like about that?

Recently I was working with a team that was having trouble getting testing tasks done, because some of the developers did not like testing; they wanted to write code, because if was more fun.

At the risk of preaching to the choir, let me explain how business works:

  1. A business employs people and asks them to make stuff. The employees would rather be doing something they would like to do, so the business pays them in order to give them an incentive to come to work.
  2. The business sells the stuff that its employees make. Customers exchange money for the stuff, because they don't have the time or the expertise to make it themselves.
  3. The business takes the money from the sales of the stuff, invests some of it in the business, pays some of it as dividends to shareholders, and pays some of it to the employees from step 1.

As long as the employees are making stuff, and the customers are buying stuff, we have a continuous cycle and a successful business.

Notably absent in the above description is "The business pays employees to come to work and only work on fun stuff."

Elbert Hubbard, a 19th century American writer and philosopher said:

If you work for a man, in heaven's name work for him. If he pays you wages which supply you bread and butter, work for him; speak well of him; stand by him, and stand by the institution he represents. If put to a pinch, an ounce of loyalty is worth a pound of cleverness. If you must vilify, condemn, and eternally disparage, resign your position, and when you are outside, damn to your heart's content, but as long as you are part of the institution do not condemn it. If you do that, you are loosening the tendrils that are holding you to the institution, and at the first high wind that comes along, you will be uprooted and blown away, and will probably never know the reason why.

Now that may sound terribly old-school. The social contract of loyalty between employers and employees is much weaker than it was even 50 years ago, when working hard your whole career almost guaranteed that the company would take care of you. (This is the social contract Billy Joel sang about in Allentown.) Today you can get laid off at a moment's notice, or your can give your two-weeks notice, and nobody raises an eyebrow.

No, this is not about being loyal to your company because it is the noble thing to do, but because if the company is paying your wages, it is in your best interest for the company to stay afloat.

There is an old story called A Tale of Three Bricklayers. (Josh Allan Dykstra has a good article about this.) It goes something like this:

A man came upon a construction site where three bricklayers were busy at work. As he meandered through the site, he asked each bricklayer what he was doing.

The first one said "I'm laying bricks."

The second one said "I'm building a wall."

The third one proudly proclaimed "I'm building a cathedral."

Developers who do not want to perform testing tasks because they are not as fun as coding are like the first bricklayer. They may be excellent coders, but they are not looking at the big picture of what the company needs in order to survive.

They are frequently long-time employees, and they are like the mice in Who Moved My Cheese?, by Spencer Johnson (one of the great agile development books, in my opinion). For as long as they could remember, there was a big pile of cheese next to their home, and when it finally ran out, some of them could not believe that their world had changed so radically, and just sat there, waiting for the cheese to come back.

There was a time, in the days of old, when being a hotshot coder was enough, and if one didn't want to help with testing or documentation, there were other departments for that. And anyway, being a hotshot coder made one close to god-like.

But the times have changed: agile development is the industry standard, and customers are beginning to expect faster response to their requests than the once-every-18-months cycle that used to be the norm.

Developers who consider themselves above tasks like coding, or ones who don't want to work on anything that isn't fun, will eventually face one of two things:

  • the company will realize that they are impeding progress and will let them go, or
  • the company will let them continue to be a spanner in the works and demoralize the developers who are trying to do agile development, and eventually they will take the company down.

Either way, they will be out of a job.