This page tells the story of a software design tool for engineers, business analysts, and domain experts, with 3 parts:
- this wiki, where all designs live
- an app, where designs can be explored and edited
- a command-line tool, where designs are transformed into code
This project, acorn, is looking for a place to mature into the transformative tool I know it can be. Read on to learn more, and please reach out at [email protected] if you see a niche acorn could fill.
After a year or two of effort I've produced something I believe structurally improves both software development and the developed software. I am, however, neither salesman nor entrepreneur born and have already explored all the options I know for commercializing this work. As no crisis should ever go to waste, job hunting during one of the ...less ideal... times in recent history gives me an opportunity to demonstrate the nature of the tool I've produced. So without further ado, let me introduce acorn.
In short, acorn is a design and documentation tool that generates functioning software systems. The central premise of this low-code platform is declarative specification of the language-agnostic elements of a back-end system. Domain-driven design is leveraged to create a common language shared by engineers, business analysts, and domain experts, while acorn avoids the trap of turning business analysts into programmers by ensuring any imperative programming is implemented using industry standard tooling. The core DNA of the tool is maturity by default without the associated lock-in of proprietary hosting or custom programming languages. This section of the wiki both explains and embodies the tool itself, so read on if I've captured your attention. Less detail can be found here.
Right off the bat - let's be clear. I didn't come up through software, though I have fond memories of making games on my graphics calculator in math class in school. Instead, my pedigree starts with mechanical engineering, a segue through biomedical engineering for a Ph.D, and finally analysis/data science to round out my little path-less-travelled. The common thread? Using my rudimentary mathematical skills to understand and influence the world around me. The fundamental issue at hand? You need software to make that happen.
Either software systems produce the data you rely on, or software systems are required to implement the outcomes. Both, as it happens, is the more likely scenario, and I rapidly found myself frustrated with the state of the software world. When presented with a new and complex domain, I want to understand:
- What data exists within the system, and what it means
- How you can interact with that data
- The pedigree of that data, and how it has evolved over time
In practice, that means attempting to digest 15 years of messy development of an organically grown system. Reading code, spelunking outdated wikis, interpreting artistic architectural diagrams that boil down to a single individual's mental model, and generally hoping that the assumptions you make are safe enough for you to rely on your own analysis. The final straw was how fraught it was to actually evolve these bespoke organic systems.
I tried to learn the professional lessons of the engineers around me that I respected, and so my journey towards acorn began.
Before I continue on, there a few concepts that set the scene. First, acorn sees three important archetypes relevant to the development of software:
- Experts: Knowledgeable about the domain, perhaps non-technical
- Engineers: The ones who make it happen, perhaps despite not knowing what it is to begin with
- E... Users: Anybody who relies on the resulting system, from non-technical users to external engineers
You management types keep the world from descending into chaos :) but these three groups need to communicate clearly for the process to be successful. As good fences make good neighbours there are three levels of formality of data access required by these groups:
- External: Highly formal contracts that have consequences when they change because external parties rely on them [Users, Experts, Engineers]
- Internal: Formal, but manageable boundaries that different sections of your organization build upon [Experts, Engineers]
- Private: Flexible boundaries enforced by a team on themselves purely for data consistency and ergonomic reasons [Engineers]
Why do these two concepts matter? Because acorn is a tool that enables those three archetypes to collaborate, and to build software that requires no extra effort to maintain the different formality levels required for safe, productive development work. Keep these concepts in mind as a lens through which to understand the meta tool that follows.
My non-traditional route makes for non-traditional nomenclature, so this section should help explain what I mean by the terms I use for acorn's meta model of a software system.
- A software system that has been designed to be embedded in other systems, as opposed to run as a stand-alone entity
- What you get when you wrap enough extra detail around a library such that it can be deployed
- The design for a library/module along with its documentation and implementation
- Individual components within a project
Thus acorn is a tool that allows you create software projects while writing perhaps 20% of the normally required lines of back-end code. These projects are standardized to allow them to be embedded (for libraries) or connected (modules) in a modular fashion. This connection could be realized as either a microservice, monolithic, or function-as-a-service deployment model. These projects themselves consist of realms, where each realm is a bounded context in domain-driven design terminology, and could be individually owned by different teams if such scale is called for.
- The basic form of data within a system, everything from raw binary information to free text or dates
- data type
- An envelope of data containing perhaps many fields containing either primitives or other data types
- A meaningful occurrence within the system, represented as a data type that is published under specific conditions
- An asynchronous process that listens for specific events and allows a system to respond to changes
- regular expression
- a custom primitive with a specified pattern
- a custom primitive that can take on a specific set of values that are defined at runtime
Thus acorn is a tool for creating data-intensive applications, where an application can be seen as a graph of primitive data fields described within data types. Editing this graph produces events at differing levels of granularity, which can be leveraged to perform meaningful responses to business-relevant changes. Oh, and realms containing data are effectively mutually exclusive sections of this graph.
A domain is the foundational realm within a system. It contains data, and provides an internal boundary for manipulating that data. There are a number of specialized concepts within a domain, and the details of the internal boundary are intended to permit standardized, flexible data access where performant, and controlled access when potentially non-performant.
- Aggregate root
- A data type that is persisted, has a defined lifecycle, and is queriable
- A named sub-graph within an aggregate root that has additional lifecycle
- A mutation operation that has side-effects and produces an event
- A read-only aggregation or transformation of data within the domain
Thus projects created within acorn store data in domains, where command/query responsibility segregation is utilized to provide a set of meaningful interactions with this information. The underlying principle behind acorn is that the internal boundary formed by these domains is necessary and sufficient for rapid development, exploratory analysis, and eventual extension.
A portal is a realm that forms an external boundary for the system. Analogous to an application service in domain-driven design, portals handle inter-domain co-ordination that goes beyond pure asynchronous event subscription. Portals contain a number of specialized concepts to create a intelligible view over the use-cases of a system.
- An externally facing set of nodes on the formal boundary of the system that share a common definition and may be associated with other contexts
- A simple query that may belong to a context
- A formal mutation operation, perhaps belonging to a context, that may return enough information to navigate to a new/existing context
- Information on how to reach different contexts from the data returned by a read operation
Thus portals, while stateless, contain all the information required to make the internal boundaries of a system both legible and safe to use for external users/downstream systems. The degenerate form of a portal is simply a layer over a domain that calls aggregate roots "contexts", commands "actions", and queries "views". In other words, a typical CRUD layer over a SQL database.
Domains support asynchronous processing, and portals form the boundary that initiates this processing. An engine is a realm that can co-ordinate asynchronous processing and solicit user input when required. Engines contain a number of specialized concepts to define state machines for these processes and thus simplify development of the inevitable BPMN-style flow charting and saga creation.
- User input
- A request for additional input, represented by an action in a portal if the request cannot be automatically fulfilled
- A logical set of background processing, starting with a command and terminating in an event after perhaps many subscriptions
- A logical saga with internal state, perhaps containing compensation mechanisms in event of failures, and containing potentially many embedded user inputs or subprocesses
Thus engines form the hand-off point between the system and the user, and contain the data required for compensation when the unavoidable eventual consistency of business leads to conflicts.
Sidenote: Eventual consistency
This castle in the sand is excessive for systems that truly just want a communal spreadsheet in the cloud (which already exists, anyway). The model described here recognizes there is more to the back-end than just SQL accessed through REST, and failing to capture this layer reduces the evolvability and interpretability of the resulting system. It also begins to chip away at the complexity introduced by native asynchronicity and eventual consistency, which I believe exists in any system that operates in the real world. Failure to cope with it locally pushes that reality to the edges of the system, where an organisation's attempt to cope with it ends up labelled "business process".
As an analogy, move the cup next to you. That was synchronous - but I saw you do it. I thought for a bit, then piped up "hey, do you want another coffee?" That was asynchronous, and eventual consistency was me coping with the embarrassment when I realized you had already started talking to someone else before I got around to opening my mouth. Moving the cup is a user synchronously interacting with their data, responding to that interaction is innately asynchronous, and managing that asynchronous, eventually consistent process is also known as doing business.
Software design and architecture can't exist purely as documentation as the inevitable drift erodes any usefulness, given time. The finished code embodies the design, but when code is the source of truth the experts and users are effectively excluded from the conversation. A new approach is required if the three groups (Experts, Engineers, and Users) want to meaningfully communicate.
First, you need a meeting point for those three groups. In this instance, you are already on it. This wiki stores those designs, and permits collaboration on the creation of said designs. If the mood calls, you can edit everything on here by hand, and much like I'm doing on this very page, can intersperse your design with explanations/justifications/miscellaneous notes. MediaWiki is an impressive piece of technology, but in reality any knowledge base with an API will do.
Second, you need a visual editor to make that design process ergonomic. Structuring the documentation like the modelled system itself helps you build an intuition while seamlessly communicating a shared understanding. That tool can be found here. The app lifts pages from this wiki, structures them in a multi-layered model, then allows you to edit the individual snippets. MediaWiki then handles conflict resolution, and if concurrent edits conflict, a merge-tool has been embedded in the app.
Third, the core functionality of the system needs to be directly created from this shared design. Anything less invites drift for all but the most disciplined, but at the same time there should be no "magic". No proprietary runtime or language, no framework with opaque behaviours, just a section of the project that is managed by this tooling. This section contains all the undifferentiated heavy lifting required to make the system's unique value available in a safe, mature, and scalable fashion. This means each of language choice, database and message queue tech, deployment model, communication protocols, and more simply become a configuration parameter in your build tooling.
In short, my intent was to provide tooling to make democratic design seamless, enshrine the agreed-upon artefacts in generated code, then get out of the way and let the engineers do what they do best - just this time with immaculate documentation. Oh, and leverage the resulting standardization to reduce cognitive load for all parties involved.
|Description||Tools and processes to aid me in my job hunt|
Note: this design can be viewed interactively with no login here
As mentioned before, I need a job. So I thought I'd build a tool to help me in that process, with the added benefit of providing a tangible example of acorn itself. This tool is an over-complicated spreadsheet that combines the required documentation with available opportunities and tracks how long each company leaves me on "read". I'll include every single snippet of hand-written code so you can see what it takes to use acorn to develop actual relevant, scalable, mature software.
Follow the guide, or explore the main realms to understand this design:
- application domain - all data storage concerns for both job applications and job opportunities
- track engine - co-ordinating asynchronous processing in the application domain with user input requests
- cms portal - providing a unified api for the underlying system
The boxes (templates, in MediaWiki parlance) on this page contain all the information required to generate the Jobhunt project. The wiki provides both a storage engine and a mechanism for expanding on this design with inline documentation. This guide introduces each of the key components sequentially, so follow the footnotes on each page to understand acorn.
This entire design created a headless CMS for my job search in 362 hand-written lines of code without making every row in the database editable from the UI. As every line is included on this wiki, you can see four key examples:
- replace version
- A command with stock-standard data plumbing
- A process with a slightly more involved lifecycle
- A view demonstrating the internal query language
- pending response
- A view demonstrating how the internal projection mechanism avoids the n+1 problem
Those 362 lines of code are now being hosted on my home NAS, and are recording everything I need to remember during my hunt for a new job!
There are a number of primitives where I know the values belong to a set, but I can't program in the values because I haven't clarified them yet. That calls for liberal use of lookups, which will eventually be provided to the finished system during installation.
This all works. I know this, because after I built the system compiler I compiled a system to make the system compiler available as a SaaS tool. Which this demo uses, and I only wrote about 20% of the resulting code by hand! So give me a job. Seriously. I can clearly program, I've had enough ops experience to run the things I've built, and I'm pretty good at math too. All I need is somewhere to direct my energy. Reach out at [email protected].
Alternatively, if you see a path to grow acorn to its full potential - please get in touch.