Project Dogfood

Monday, January 19th 2009 | Ismael Ghalimi

While the current recession is forcing many companies to go through painful rounds of layoffs, Intalio is actually growing its employee base, albeit at a slower rate than in the previous twelve months. There are two reasons for this: Number one, we want to remain on the conservative side, and continue outsourcing most non-core functions, which pretty much means everything outside of architecture design and customer training. Number two, we have started to aggressively use our own products in order to improve our operations, thereby allowing us to do more with less. We call this initiave Project Dogfood.

WARNING: very long article, but full reading highly recommended.

Project Introduction

The idea for Project Dogfood emerged at a recent company meeting, and was refined during our yearly company offsite. The idea is pretty simple: every single employee (36 out of our 60+ full-time collaborators) will develop a new process, or improve an existing one, using our products. The project is managed as an internal competition, and the rules are relatively straightforward:

  • 12 teams of 3 employees must be formed.
  • Each team must include at least one engineer and one process expert (trainer).
  • Each team must design and deploy at least one real-world business process.
  • No two teams can implement the same processes (first come, first served).
  • Processes must contribute to improve Intalio’s operations.
  • Processes should not use confidential information (publishing encouraged).
  • Teams must use off-the-shelf Intalio software.
  • Any product edition can be used (combinations encouraged).
  • All custom code must be deployed as a J2EE webapp or XPath function.
  • Custom product features must be developed through the D3 process.
  • D3 projects must be funded by external customers, or actual cost savings.
  • D3 projects funded internally must be approved by the CFO.
  • Projects should have a business sponsor, who does not have to be part of the team.
  • Processes must be deployed on the latest versions of Intalio’s products.
  • Processes must be used in production on an ongoing basis.
  • Processes must be kept running when new product versions are deployed.
  • New product versions cannot ship until all processes have been upgraded.
  • Teams must capture time spent working on the project.
  • Teams must create a business case before implementing the process.
  • Teams must demonstrate Return on Investment after deploying the process.
  • Teams should publicly communicate about their projects (website, blogs, etc.).
  • Teams must write a detailed success story presenting their projects.
  • All teams must be formed by January 31, 2009.
  • All projects must be introduced by February 28, 2009.
  • All projects must be completed by June 30, 2009.

Once all projects are completed, they will be presented to our Board of Directors, which will select the winning project, based on the following criteria:

  • Best success story
  • Highest Return on Investment
  • Best utilization of Intalio’s products
  • Highest respect of industry best practices
  • Most contributions to product improvement

The winning team will receive $10,000 in Amazon credits, to be shared equally among all 3 teams members. The CEO is eligible to participate, but shall donate his prize to a charity of the team’s choosing in the event that he would be part of the winning team. The prize will (hopefully) be funded by savings realized from the deployment of processes developed through the project.

Sample Process

Since I am eligible to participate, I have decided to automate the process that could have the most impact on Intalio’s success in 2009: our Marketing & Public Relations process. Historically, marketing has been Intalio’s weakest points, and our resources are extremely limited, with only one full-time employee in our marketing “team”. Our marketing budget is essentially spent on payroll and a very modest use of Google AdWords (about $4,000/month). We do not go to tradeshows, we do not advertise in trade publications, and we do not retain the services of a PR agency anymore. That’s what I call marketing on a shoestring budget.

But lack of resources should not be an excuse for lack of execution. With today’s tools, there are many things that can be done with very limited resources, as long as you can put a good process in place, and this is what my team is about to implement. Talking about the team, I have the privilege of working with Antoine Toulme (Blog), lead developer for Intalio’s BPMN modeler (based on Eclipse BPMN Modeler, which we contributed three years ago), and Rick Geneva, process expert based in Atlanta, GA, and co-author of the book The Microguide to Process Modeling in BPMN.

Process Discovery

The process we are implementing is responsible for keeping track of all marketing and public relations activities that must be conducted in response to newsworthy events, such as the release of a new product, the acquisition of a new company, or the hire of a new executive. Using Google Docs (see spreadsheet in .xls format) for process discovery, we identified 25 such news item, and defined the set of actions that need to be conducted in response to them. We identified 20 such steps, including drafting marketing collaterals, conducting analyst briefings, or issuing a press release.

Since the downstream process is actually fairly linear, we used the spreadsheet editor as a way to capture all process steps, in a two dimensional way: news items are listed in rows in no particular order, and steps are listed in columns in sequential orders. Steps were then grouped in sequential phases (using cell merging, also known as colspan in HTML). Steps that are part of the same phase can be executed in parallel. We identified seven phases, plus the process instantiation phase (Phase 0).

Once all news items, phases, and steps were identified, time came to fill up the matrix, indicating which steps were required for every news item. For doing this, we used the following conventions:

  • ‘X’ sign: mandatory step
  • ‘?’ sign: contextual step
  • ‘O’ sign: optional step

Contextual steps are not mandatory and are included in the list of steps to be completed for a given news item depending on the context of the news item. The decision to make a contextual step part of the process instance is taken by the Marketing Manager during the news item’s initial review (Pase 1/Review Item). Optional steps are always included in the process instance, but their completion is not required, and the process instance can continue its execution if the optional step has not been completed within a configurable time period.

Completing the matrix of 500 cells gave us a full map of the process. If the process were to be mapped in BPMN directly, it would require 500 decision gateways, but using the matrix to feed a rules engine will reduce this number down to 20.
Once this work was completed, we defined the Roles who would be involved in the process, and identified 5 of them:

  • Vice President of Alliances
  • Vice President of Marketing
  • Marketing Manager
  • IT|Redux Editor
  • Contractor

Process Implementation

We then specified which notification mechanism would be used for each step, based on the preferences of each Role holder: the Vice President of Alliances is using Microsoft Exchange and a BlackBerry device, and wanted tasks to be created directly into Microsoft Exchange; the Vice President of Marketing (a role I am holding now) is doing pretty much everything withing, and wants all his tasks to be created there; the Marketing Manager (Jonathan, who really holds the title of Director of Marketing, but it did not fit into the spreadsheet’s cell) offered to use Intalio’s Task Manager; the Editor of the IT|Redux blog is the same as the Vice President of Marketing (damn, I am wearing a lot of hats…); finally, the Contractor is also using Intalio’s Task Manager, thereby demonstrating that such a component can be used outside of the firewall.

Once all that was set, we defined how each step would be implemented. This allowed us to identify 6 types of step, that would be packaged as composable services:

  • Data Input Task, with a form to be produced using Intalio|AJAX
  • Approval Task, with three possible actions (Approve, Deny, and Comment)
  • Manual Task, with two possible actions (Complete and Save)
  • Multiple Tasks, which are nothing more than multiple Manual Tasks
  • Email, sent to a small number of recipients (typically less than 100)
  • Email Blast, sent to a large number of recipients (typically up to 100,000)

At the exception of the Data Input Task and Multiple Tasks steps, most steps are pretty generic. Nevertheless, we felt the need to provide more details for some steps, such as the Email and Email Blast steps. For them, we indicated recipients to emails directly in the cell, sometime suffixed by the ‘?’ sign in order to indicate a contextual step.
For the Data Input Task step, we listed the combined set of form elements used by forms to be displayed to the Marketing Manager during the initial review step (Phase 1/Review Item). For most form elements, we marked the required ones with the ‘X’ sign. In some cases, we provided more details, such as the expected content of the Description field (Biography, Executive Summary, etc.). For Links (URLs), we indicated the content of the linked page (Datasheet, Use Case, etc.), and the online service expected to be used for serving it (WordPress for example). We also did the same for attachments, indicating the software to be used for producing them (Apple Keynote or Microsoft Word for example). We also suffixed all form element labels with their datatypes, like [Text], [TextArea], or [URL]. This essentially gave us all the information we needed to dynamically produce contextual process instantiation forms.

In order to complete our process design, we specified which kind of event would be responsible for creating a new process instance, and used comments on cells to indicate who in the company was expected to generate such events (in this particular case, comments are used for process documentation purposes). We identified four main kinds of process instantiation events:

  • Web Form (WF)
  • RSS Feed (RSS)
  • Google Alert (GA)
  • Alert (SFDC)

At this point, we had a complete model of our process. High-level modeling took our Vice President of Sales & Operations (business sponsor), Jonathan, and I about 1 hour (3 person-hour). Low-level modeling, including all deployment details presented in this article took me another 5 hours or so (I finished the spreadsheet and wrote this article on my way to Tokyo, for movies available on my flight were simply too bad to waste time watching them — I won’t give titles).

Framework Design

What came next is the really interesting part of this article, and justifies the aforeindicated (neologism?) warning. When I say that we had a complete “model” of our process, I should say that we had a complete “design” instead, for everything needed to make this process executable was right there, in the spreadsheet. Assuming that conventions would be respected, all the information needed for making this process executable had been captured in the spreadsheet, using what is arguably one of the most primitive spreadsheet editors currently in use today — albeit a very convenient one, which I use for most of my spreadsheets, on a daily basis, thank you Google!

The challenge then became: how do we turn this spreadsheet into an executable process by using off-the-shelf software and writing a minimal amount of code, using only publicly available APIs? Furthermore, how could we allow the process to be changed directly from the spreadsheet editor, and support dynamic process re-deployment? After a few rounds of discussions, we figured out a working strategy, which will require the following tasks:

Task #1: Implement Composable Services
As mentioned above, we identified six composable services, which can easily be implemented using Intalio’s connectors and workflow framework. These services will be packaged as WSDL Web Services, and deployed on Intalio|Server. Furthermore, they will be registered into Intalio|ESB, so that other projects could take advantage of them. Here is how each service will be implemented:

  • Data Input Task: forms will be produced for Intalio|AJAX from XML Schemas
  • Approval Task: standard workflow task (Comment implemented using Save)
  • Manual Task: standard workflow task
  • Multiple Tasks: multiple standard workflow tasks created within a FOR-EACH loop
  • Email: standard Email connector
  • Email Blast: custom connector for VerticalResponse

By default, Approval Tasks and Manual Tasks should display all the data captured through Data Input Tasks. Furthermore, Approval Tasks should support data editing as well. Approval Tasks should also make it easy to change or edit attached documents. Google Apps should be used for the online editing of attachments which formats are supported by the application.

Task #2: Create a Form Generator
The form used for initial review (Step 1/Review Item) has 25 variations, one for each news item type. This form will be generated automatically from an XML Schema, which itself will be generated automatically from the master spreadsheet (later referenced as master spreadsheet).

Unfortunately, while Intalio|AJAX supports the development of contextual forms for which some elements are optionally displayed based on some specific context, the XML Schema to Form converter does not support this feature yet (Feature Request). In order to work around this limitation, one form will be produced for each news item type.

In order to do so automatically, the spreadsheet will be published as a .csv feed, and a web application developed in PHP (a language I am familiar and comfortable with, unlike Java) will generate an XML schema for each news item type. This PHP application will run on top of Quercus, a Java-based implementation of the PHP language. The reason for this is the following: Intalio|Server does not support the execution of PHP code out of the box, and all custom applications must be deployed as a Java-based webapp (per the role of our competition). Fortunately, Quercus can be packaged as a webapp, and serve as container for the execution of custom PHP code. In the future, I would expect Intalio|Server to ship directly with the Quercus PHP interpreter, as well as Jython for Python and JRuby for Ruby (Feature Request). The PHP script will produce a set of XML Schemas to be parsed at deployment time by the XML Schema Import Wizard component of Intalio|Designer, which itself will be deployed as an OSGI runtime component alongside Intalio|Server.

Task #3: Create a Decision Table Generator
The spreadsheet contains all the information for the decision table (AJ7:BA31) to be fed to the business rules engine (Intalio|BRE). The decision table will be converted into a smaller .csv document using another PHP script, and deployed on the business rules engine using a new API to be provided by Intalio|Designer. This API will be developed through a Demand Driven Development (D3) project for which we are currently looking for sponsors.

Task #4: Create a BPMN Generator
This is where things get really interesting. The idea is to automatically generate the BPMN process map from the first five rows of the spreadsheet.

  • Row 1: Phases, modeled as sequential sub-processes.
  • Row 2: Steps, modeled as parallel sub-processes.
  • Row 3: Roles, modeled as independent swimlanes.
  • Row 4: Implementations, modeled as sub-processes invoking composable services.
  • Row 5: Task Management, providing details on the composable service to be used.

The process will be modeled through one swimlane responsible for orchestrating the end-to-end flow, plus one swimlane per Role. Each service invocation sub-process (Implementation) will be modeled as a an activity preceded by an invocation of the rules engine and a conditional gateway, which together will be responsible for defining whether the corresponding step should be part of the process instance or not. This pattern is equivalent to the concept of pre-condition for activities, which is not part of the standard BPMN specification. Nevertheless, proper use of sub-processes should hide this implementation detail, and depict each step as a single activity.

The generation of the BPMN code will be done by another PHP script deployed on the Quercus runtime, on top of Intalio|Server. This script will translate all the information related to the process flow (B1:BA5) into a BPMN serialization format contributed by Intalio to the Eclipse BPMN Modeler project. Even though not a standard yet, this format is perfectly human readable (unlike XMI), and has the benefit of being available today. The PHP script will produce a serialized BPMN document to be parsed at deployment time by a server-side implementation of Intalio|Designer. Unfortunately, the BPMN parser currently does not support references and binding to web services, therefore will have to be extended through a D3 project for which we are currently looking for sponsors.

Task #5: Implement a Dynamic Deployment Process
A Dynamic Deployment Process will be implemented using a custom BPMN process. This process will be instantiated automatically every 5 minutes, and will lookup the .csv feed provided by the master spreadsheet. If the .csv feed has been modified since the last time a new process version was deployed and the value of cell A1 is “Status: Deployment”, the deployment process will continue its execution. Otherwise, it will terminate. By convention, the developer of the process should change the value of cell A1 to “Status: Development” when making modifications to the process.

If a new version of the .csv feed is available, the Dynamic Deployment Process will invoke the Form Generator and deploy a new set of forms on Intalio|AJAX, invoke the Decision Table Generator and deploy a new set of rules on Intalio|BRE, and invoke the BPMN Generator, import the BPMN process map, generate the corresponding BPEL code, and deploy a new set of BPEL processes on Intalio|Server. The Dynamic Deployment Process will then send an email to the process owner notifying her that a new version of the process has been generated and deployed.

Expected Product Improvements

If we manage to make it all work, we will have developed what might be the first spreadsheet-driven BPM tool, usable by any business analyst, and available as Software as a Service. As currently designed, it could support any process made of sequential phases comprised of multiple parallel steps. Nevertheless, more complex flows could be designed from the spreadsheet by using a construct equivalent to GOTO. In order to model such an arbitrary transition from one step to another, the user could reference a Phase to transition to by simply indicating the Phase’s column reference in the appropriate cell.

Ten years ago, I suggested that BPM tools should do for process execution what spreadsheet editors did for computation. With this application, we’re about to make using Intalio’s BPM tools as easy to use as using a spreadsheet editor, literally. And the side benefits of the approach are quite interesting as well:

Web 2.0 Collaboration
By using an online spreadsheet editor such as Google Docs, we make it easy to share executable process models among collaborators, or publish them on the Internet for anyone to see. Access rights can be easily granted and revoked by simply adding and removing email addresses to and from a list of collaborators, across organization boundaries. Additionally, tools such as Google Docs provide full versioning capabilities, making it easy to keep track of all changes made to process definitions and revert to previous versions.

Since the entire process model is designed within a spreadsheet, all parameters and rules driving the process execution can be the result of complex calculations executed at deployment time. Such calculations can be implemented as simple formulas when using any online spreadsheet editor, or complex ones making invocations to macros when using more sophisticated editors such as Zoho Sheet.

Integrated Dashboard
Online spreadsheet editors such as Google Docs or Zoho Sheet allow the content of cells to be dynamically updated through Web Service interfaces. Using this feature, the process engine (Intalio|Server) could feed metrics and KPIs back to the master spreadsheet for real-time Business Activity Monitoring (BAM). As a result, process owners would have a fully integrated interface for defining the process, setting business rules, monitoring the process in real-time, and applying process optimizations on the fly, all from the intuitive spreadsheet metaphor that any business user is accustomed to.

Service Composition
While all the capabilities listed above are interesting, the most important by-product of this project should be a better understanding of how Intalio’s products can support the notion of Service Composition. Currently, Intalio|Designer allows processes to be packaged as reusable services, but requires that data mappings be specified at every step of the process. While the Data Mapper component of Intalio|Designer makes it easy to design both simple and complex mappings, the very requirement for them gets in the way of simple Service Composition that could be performed by non-technical users.

The meta-data driven approach advocated by our project intrinsically relies on the availability of composable services designed against pre-defined data schemas and packaged with all required mappings. While such an approach makes it a little bit more difficult for a composable service to leverage a new data element defined for a Data Input Task in the master spreadsheet, it makes it a lot easier for new processes to be composed from existing services. But for such an approach to work , composable services must provide the right set of interfaces and built-in mappings, and much work remains to be done in order to discover what such interfaces should be to address a wide range of scenarios. Once such a discovery is made, many different metaphors could be used for supporting the notion of Service Composition, and the spreadsheet is only one of them, albeit possibly the most intuitive and flexible.

Expected Business Benefits

With respect to the actual Marketing & Public Relations process, the benefits we are expecting are many. First among them is the ability for our small marketing team to scale its output as our business is growing, without having to increase the size of the team, nor using the services of outside contractors. Properly handling our PR needs would in and by itself save us about $200,000 a year. Implementing the process as designed should also make it easier for some steps to be outsourced, such as lead generation for local events for example (Phase 3/Generate Leads). Second, we expect that by simply issuing a steady stream of press releases, blog posts, and white papers, we will increase Intalio’s visibility in the marketplace, thereby increasing the number of visits to our website. Third, by providing up to date content on the website, we expect conversion rates from visitors to users and users to paying customers to increase as well. Finally, by automating the end-to-end Marketing & Public Relations process, we hope to gain better visibility on the effectiveness of fine-grain marketing campaigns (such as the distribution of a Press Release or the publishing of a White Paper), which will allow us to better allocate our limited marketing resources.

Ongoing Updates

Our intent is to fully document our progress online. We will record the time spent working on the project on a timesheet shared among collaborators, alongside new ideas and integration requirements captured through additional sheets on the master spreadsheet. We will publish regular updates on this blog and a section of the new Intalio website to be added soon. We will release our code under the Apache Public License, and share as many process artifacts as possible (master spreadsheet, .csv feeds, XML Schemas, generated forms, decision table, Serialized BPMN documents, generated BPMN maps, WSDL interfaces, etc.). We will also publish real-time metrics regarding the occurrences of news items and process cycle times once the process is deployed.

Future Roadmap

If we are successful, we will package the Form Generator, Decision Table Generator, BPMN Generator, Dynamic Deployment Process, and all generic composable services (Data Input Task, Approval Task, Manual Task, Multiple Tasks, and Email) as a process template, and make it part of Intalio|BPP Business Edition, Intalio|Community Edition, and Intalio|BPP Enterprise Edition.

Wish us luck, and consider sponsoring some of our D3 projects!

Entry filed under: BPM 2.0

5 Comments - Add a comment

1. Dmitry Gusev  |  January 20th, 2009 at 12:39 am

Thanks, Ismael!
That’s a really great article!

2. Apache Sling, Jackrabbit,&hellip  |  January 23rd, 2009 at 11:06 pm

[…] I was playing around with it, and decided to make a quick contribution to tempo as part of the Intalio dogfood project. […]

3. IT|Redux - A first Taste &hellip  |  January 25th, 2009 at 12:37 pm

[…] Project Dogfood was launched just two weeks ago, but it’s running full speed ahead. Among the most active […]

4. Adam Sah  |  January 30th, 2009 at 4:06 pm

Awesome stuff, on multiple levels — case study, dogfooding, transparency, etc.

Hats off!

5. IT|Redux - More on the Ta&hellip  |  February 12th, 2009 at 7:05 pm

[…] process patterns I found particularly interesting. It turns out that I am not the only one, and a Dogfood team has been formed around it, with Robert (Vice President of Business Development) and Madhav […]

Trackback this post  |  Subscribe to the comments via RSS Feed

Leave a Comment


Required, hidden