Category: General

General thoughts that defy categorization.

Big changes ahead

Austin skylineI’m making some big changes, both professionally and personally, that I’m very excited about. First, I’m shutting down Metaversant at the end of this month. I started the company a little over 12 years ago and I’m sad to see it go, but it is time for new challenges and opportunities.

It was exhilerating striking out on my own. I started with nothing but a laptop, a passion for technology, and a reputation for doing great work, and I created a profitable business delivering solutions for well-known organizations like NASA/Jet Propulsion Laboratory, Southwest Airlines, Red Hat Software, and Duke University.

But I’m also proud of the work I did for lesser-known clients, many of whom were active customers for six or seven years. Sure, I had a few “transactional” engagements here and there, but for several organizations I became a trusted advisor and developer for all kinds of technical endeavors for multiple years.

Many of you know me as an “Alfresco guy” and I’ve certainly benefitted tremendously from the deep knowledge I developed around that platform and from the wonderful people in the Alfresco community that I helped foster and grow. I wouldn’t trade those experiences or the relationships I built with people in all parts of the world for anything.

At the same time, some of my most exciting, rewarding, and impactful projects have been in other areas of content management and custom application development. Projects where code is just flying from my fingertips while I’m in the zone, or projects with many different systems working together in the cloud, or projects built collaboratively with talented teammates delivering solutions with millions of users. Those projects get me out of bed in the morning.

I’m someone that needs constant challenges and a lot of variety in what those challenges consist of. While starting my own company was a life-long dream, the kinds of projects that find me aren’t as exciting as they used to be. Yes, I could stop taking certain kinds of projects and put more effort behind marketing and business development for the projects I’m interested in. Or I could join a team that is awash in projects like that. I chose the latter. At the end of the month I’ll be joining Apple as a Principal Architect working in the Content Management Systems Center of Excellence.

As you can imagine, Apple has all sorts of content management challenges. A lot of those revolve around music and video so that appeals to me a lot. Regardless of the type of digital experiences their CMS technologies support, doing so at “Apple scale” really excites me.

As part of the new gig, Christy and I are moving to Austin. With its thriving live music scene it’s always been my kind of town, but as it has continued to grow as a tech hub it really became impossible to resist. And, I already have a fairly serious breakfast taco habit, so that’s one thing that won’t have to change with the relocation.

The geography change is kind of like the job change–we’ve lived in the same area for thirty years and it is definitely time to shake it up, especially with a mostly-empty nest. Staying in Texas wouldn’t necessarily be our first choice, but we both love Austin and it is close enough to Dallas that we’ll be able to visit friends and family quite frequently.

I want to thank all of the customers I’ve served over the years. We did some really great work together and we had a lot of fun doing it. I also want to thank everyone in the Alfresco community for reading and sharing my writing and my code, for sharing your knowledge and insights with me, and for encouraging me to keep going. How boring it must be for the unlucky ones who spend their days implementing solutions on platforms that don’t have such an amazing group of friends and colleagues scattered around the globe, collaborating with each other for the benefit of all.

Backpacking trip: Hoh River Trail to the Blue Glacier

I’m back from ten days in Washington State. I set the girls up in an Airbnb in Tacoma then headed into the wild with my long-time friends for a six-day backcountry adventure. We hiked 36.2 miles along the Hoh River Trail to the Blue Glacier and back, which is in the Olympic National Park. I’ve been to Seattle a bunch and have explored the San Juans by sailboat (see here and here), but this was my first trip to the Olympic National Park.

This trip had multiple milestones for me:

  • Longest total miles in a single trip. Total altitude was pretty tame, although I don’t have the specifics.
  • First time using a hammock tent. Luckily, there were plenty of trees at every campsite!
  • First time using trekking poles. Wow, total game-changer!
  • First backcountry trip where I didn’t get lost. (The time-I-got-lost-on-Mt.-Rainier story is a crowd favorite. I need to post it at some point).

We came back with the same number of folks we started with, and all reasonably healthy, but tired. I remember at the end of the trailhead telling the team I felt like I could do another week. Then, once I sat around a bit, the fatigue caught up with me and I’m still recovering.

The glacier was beautiful, though, and I really enjoyed that focused time with my friends.

If you want more details, check out:

Adding actions to the Alfresco Share Selected Items menu

In a typical Alfresco implementation it is quite common to develop one or more custom actions to encapsulate a set of operations that need to be performed against a folder or document. Custom actions can be called from folder rules, added to the user interface, or called programmatically from web scripts. To learn more about developing custom actions, check out my tutorial on custom actions.

When actions are added to the Share document library or document details page they operate against a single folder or document. But the Share document library also lets users select multiple objects and then run actions against them. Out-of-the-box examples are things like “Download as Zip”, “Copy”, “Move”, “Start Workflow”, and “Delete” as shown in the screenshot below.

Alfresco Share out-of-the-box selected items menu

This blog post explains how you can add your own custom actions to the “Selected Items” menu in Alfresco Share.

First, I’m going to assume that you already have a custom action that works when invoked from a rule or from the UI and you just need it to work from the Selected Items menu. I’ll call mine “Example Action”. It doesn’t do anything exciting except log the node reference of the object the action is run against. Here it is sitting in the list of document actions on the document details page:

Custom action displayed in the Alfresco Share document details actions list

Now the goal is to add Example Action to the “Selected Items” menu so that the action will run against each object the user has selected. If you want to skip right to the working project on GitHub that’s fine, otherwise, read on for a step-by-step…

Step 1: Extend the multi-select configuration

I used a Share extension to add some custom configuration to get Example Action to show up in the document library browse list and the document details page. I’ll add additional config to extend the out-of-the-box “multi-select” list of action links:

<config evaluator="string-compare" condition="DocumentLibrary">
    <multi-select>
        <action type="action-link" id="onExampleAction" label="actions.example-action"></action>
    </multi-select>
</config>

The action label is a property bundle key that has the text I want to appear in the menu. The action ID, onExampleAction, is the name of a client-side JavaScript function to be called when the user selects the menu item.

I’ll implement the onExampleAction function in a file named custom-actions.js. I need to get Share to include that file as a dependency, so I add an additional config block:

<config evaluator="string-compare" condition="DocLibCustom">
    <dependencies>
        <js src="resources/multi-select-demo-share/custom-actions.js"></js>
    </dependencies>
</config>

That’s all I need to change in my Share extension XML file. See the full file on GitHub.

Now let’s look at the client-side JavaScript.

Step 2: Implement the client-side JavaScript

In the previous step I configured Share to add an action to the “Selected Items” menu that invokes the onExampleAction function when clicked and I told it where to find the function. Now I need to implement the function. To do that, I use YUI to fire a “registerAction” event, passing in an object with two properties: one is the name of my action and the other is a function to execute:

(function() {
      YAHOO.Bubbling.fire("registerAction",
          {
              actionName: "onExampleAction",
              fn: function custom_DLTB_onExampleAction(assets) {
                  // Function logic goes here
              }
          });
})();

Note that “onExampleAction” matches whatever I specified as the action ID in the multi-select config. It has nothing to do with the name of the Java class that implements the back-end action executer.

Within that function body you could do whatever you want. Maybe you need to capture some additional data from the user, so you present a modal. Maybe you want to display a progress bar. It’s up to you.

In my case, I just need to send the selected nodes to the back-end. Remember that I already have a working action. Mine is a Java class named ExampleAction that writes a log message with the node reference of the node the action is running against.

It would be nice if I could just somehow call that action and hand it my list of selected node references, but the execute method in the ActionExecuter interface only accepts a single node. Plus, we’re currently sitting in client-side JavaScript delivered by Share that needs to invoke an action in the Alfresco back-end–a completely separate web application. I need a way to asynchronously make an HTTP call to the Alfresco back-end from here.

The solution is to call a custom repository-tier web script with the list of selected node references, and use the web script controller to invoke the action for each of the node references in the list.

Alfresco provides a utility for making asynchronous posts and a constant that has the Share proxy URI. (The Share proxy avoids making the call from the browser all the way back to the Alfresco back-end, which could be running on a different host.) All I have to do is provide the path to the web script and the list of assets:

Alfresco.util.Ajax.jsonPost(
    {
        url: Alfresco.constants.PROXY_URI + "ecmarchitect/example-action-multi",
        responseContentType: "application/json",
        dataObj: assets,
        successMessage: this.msg("message.example-action.success"),
        failureMessage: this.msg("message.example-action.failure"),
        successCallback:
            {
                fn: function exampleSuccess() {
                    console.log("Success");
                },
                scope: this
            },
        failureCallback:
            {
                fn: function exampleFailure() {
                    console.log("Failure");
                },
                scope: this
            }
});

In the above JSON POST I am also setting a success message and a failure message so that the Share user sees feedback after the web script is invoked. Just to show you how it works, I’m also including success and failure callback functions. Mine just write a message to the browser developer console, but you could do whatever you need to, such as display data that came back in the web script response.

You can review the full source of custom-action.js or go on to the next step, which is to implement the web script.

Step 3: Implement the web script

I need a web script that will accept the list of selected nodes and invoke the action for each one. Webscript controllers can be written in either JavaScript or Java (tutorial) but I almost always write them in Java these days.

You can see the full source of ExampleActionMulti.java on GitHub–I am just going to include the relevant snippet here:

Action exampleAction = actionService.createAction("example-action");
JSONArray dataObj = (JSONArray) content;
for (int i = 0; i < dataObj.length(); i++) {
    JSONObject obj = (JSONObject) dataObj.get(i);
    NodeRef nodeRef = new NodeRef((String) obj.get("nodeRef"));
    if (nodeService.exists(nodeRef)) {
        actionService.executeAction(exampleAction, nodeRef);
    }
}

Alfresco Share handed my client-side JavaScript function an array of node objects which I then post to this web script. So the web script iterates over that array and grabs the node reference string to construct an actual NodeRef object. I use the node service to make sure that node ref actually exists, then I execute the action against it.

Other parts of the web script that I’m not showing here are the web script descriptor, the web script view template, and the Spring bean that injects the node service and action service dependencies into my controller.

Step 4: Test

With everything in place, I’m ready to test. My project was bootstrapped with the Alfresco SDK, version 4.1.0, so it uses Alfresco Community Edition 6.2.0 by default and runs locally using Docker. After running run.sh build_start I can log in to Share, go to a site’s document library, selected multiple objects, and see this in the Selected Items menu (click to enlarge):

Alfresco Share selected items menu showing the custom action

And I can see this in the log:

Log output showing that the action worked

That’s it. Now you know how to add your own custom actions to the Selected Items menu in Alfresco Share.

My Proposal for Resurrecting the Alfresco Add-ons Directory

Friday, May 25, 2018 was an auspicious day. You may remember it as the crescendo in the wave of GDPR-compliance related email notifications because that was the day everyone was supposed to have their GDPR act together. For those of us in the Alfresco Community, it was also the day that the Add-ons Directory died (or, more correctly, was killed). In fact, the two are directly related. More on that in a minute.

First, a quick disclaimer: My team and I created the Add-ons Directory back in 2012 when I worked for Alfresco. I also have multiple Add-ons listed in the directory.

But this post isn’t about being upset that something I created went away. I’m not sentimental about such things. Instead, I want to give some context as to why an Add-ons directory is important, grumble a bit about its abrupt demise, and, more importantly, provide a vision for how we can restore what many of us considered to be a valuable community asset.

Why an Add-ons Directory is Important

From a practical standpoint, I referenced the old Add-ons directory on a weekly basis. Customers and community members would routinely ask about me about functionality like e-signatures, scanning, and multi-factor authentication, and it was useful to be able to point them to the directory to learn more.

From a bigger-picture perspective, an Add-ons directory does the following:

  1. Helps customers extend the platform to close requirements gaps without requiring them to write code or hire developers by making it easier to find existing Add-ons
  2. Matches Add-on seekers with Add-on providers, some of whom might have a hard time getting noticed depending solely on organic search
  3. Provides a mechanism for the community to give public feedback on Add-ons which the developers can incorporate to improve their offerings and the rest of the community can use to help judge whether or not a particular Add-on meets their requirements
  4. Can form the foundation upon which a marketplace can be built to facilitate monetization of Add-ons, trading value (dollars) for resources (useful, quality extensions)
  5. Acts as one potential proxy for the vitality of a developer ecosystem

Over time, customers tend to have very similar requests for Alfresco customizations. Often, the community addresses this by building Add-ons and making them available either commercially or as freely-available, open source-licensed downloads. Without a central directory, the only way to find these is by searching the internet or through word-of-mouth. A directory of Add-ons provides a way to shortcut that search. When combined with social features like comments and up-votes, the community can help spotlight the most useful Add-ons.

A directory can also serve as the foundation for a marketplace. An application marketplace is more than a directory, but it cannot function without one. The old Alfresco Add-ons directory included an RSS feed that the product subscribed to and displayed in the Share user interface which helped give visibility to the directory and newly-created Add-ons, but that’s as far as it got toward being a marketplace. We had a vision that included being able to search for and enable Add-ons from within the Administration panel, but the architecture of the platform and engineering priorities kept that vision from being achieved. We never got anywhere close to being able to facilitate payment for those that wished to do so.

Despite being just a directory, when we launched Add-ons, we watched the number of submissions grow fairly steadily. There were over 500 listed when the site was taken down. This growth motivates everyone in the community: Developers see an ecosystem in which they can participate to market their products and services; Sales people see a way to help close deals by pointing to the activity around the platform and the ability for the customer to close requirements gaps with Add-ons; and Customers are reassured that there really are developers who are investing time and talent in the platform.

Why it Went Away

Alfresco says that GDPR concerns forced them to take down the site. You can read more on that here and here. Assuming the GDPR reasoning is valid, it is a bit irksome because GDPR was a surprise to exactly no one. If the site was a GDPR compliance problem, that should have been identified early enough to put a plan in place for a calm migration to a suitable alternative with plenty of advanced notice to the community. Instead, the site was abruptly shutdown that Friday with hastily-crafted blogs posted after-the-fact followed by a weeks-long effort to restore at least part of the data.

That effort is now complete and the listings are available in Alfresco’s community site here. I’m glad the data was restored, but this can’t be considered a long-term solution because:

  1. When searching for an Add-on you must now wade through results that include both Add-ons and forum threads.
  2. The ability to perform a faceted search is limited to filtering on release.
  3. The old Add-ons RSS feed that is used by the Add-ons Dashlet in the product is still broken and fixing that would be painful because in the community site, each Add-on is now simply a blob of unstructured, static text.
  4. This unstructured listing is painful for developers to maintain and ill-suited to programmatic access, which would be necessary to support an actual marketplace.

Clearly something different is needed and I’ve got an idea that might be the start of something kind of interesting.

A Proposal for Moving Forward

The first thing we need is a way to describe an Add-on that is:

  1. Independent of the directory that indexes it
  2. Owned and managed by the developer
  3. Easily updated via script

The problem of using a generic project descriptor and then ingesting that into a public directory has already been solved. Java projects have a POM file that includes metadata about the author and a description about what it does. That file can be configured to make it easy to publish to Maven Central, a public directory of Java packages. The Node world has package.json and the NPM registry. In Python you create a setup.py file that the public Python Package Index can read.

The Alfresco world needs something similar, so I’ve created a project on Github called alfresco-addon-descriptor. The project contains a JSON Schema for describing an Alfresco Add-on, an example addon.json file that adheres to the schema, and a runnable Java class that validates a JSON file against the schema.

This is a small start of something that could grow. I’m open to suggestions, but here’s how I see it potentially playing out…

Step 1: Get the schema roughly right

I took a swing at the schema but I want your feedback. I don’t think it needs to be perfect–we can version it as it evolves. We just need to get it roughly right for now. Please create a pull request if you have ideas for improvement.

Step 2: Everyone puts addon.json in the root of their project

Once we get a decent draft of the schema in place, we get everyone to put addon.json in the root of their Alfresco Add-on projects.

Now, the description of an Add-on is much “closer” to the developer. In fact, keeping it updated could be scripted if the developer so chooses. This will cut down on directory entries being out-of-date.

If we go no further, at least we’ll have a standard way of describing an Alfresco Add-on that could be leveraged by code and directories in the future.

Step 3: We stand up a simple directory that indexes everyone’s addon.json files

I am happy to do this and would probably run it under the ecmarchitect.com domain. But the beauty of the approach is that anyone that wants to could stand up a directory, all fed by up-to-date entries stored where it makes sense–in the projects themselves.

Developers could submit pointers to their files once, and the system would poll over time to keep them updated. The directory wouldn’t be able to poll closed source projects, but developers could just submit their updated addon.json file whenever it changes.

If I run the directory, I might charge closed-source Add-ons a nominal yearly listing fee to subsidize the operational costs of the directory similar to how Github can be used for free by open source projects while others have to pay a little bit.

If we stopped here, we’d have a more up-to-date and functional Add-ons directory with faceted search and a RESTful API.

Step 4: Support for monetization

Suppose this idea takes off and this new distributed directory of Alfresco Add-ons gets populated by open source and closed source developers alike. The next step is to make the directory a true marketplace.

I know that not everyone wants to charge for software and that some are morally opposed to software licenses. The directory would continue to support open source Add-ons as first-class citizens from day one.

For those that do want to charge for their Add-ons, actually doing so can be a pain. What if it was as easy as describing your cost model in your addon.json file and doing a small one-time registration in the Add-ons site to work out payment details?

On the customer side, what if you could search the directory, then pay for your Add-on subscription with a credit card?

This would make it easier for small and independent developers to get paid, and would remove paperwork and procurement drudgery for everyone.

It is unlikely that Alfresco will ever provide such a service due to the accounting, development, and infrastructure involved, not to mention possible messiness around partner conflicts, supporting a payment stream for non-partner community developers, etc. Plus, it’s just not core to their business of providing content and process services. But an independent directory would have no problem pulling this off. It’s not a big money-making idea, to be sure, but that’s not the point. The point is to restore a community asset, making it better than it was before.

If we make it this far, we will have resurrected the old Add-ons directory and added more features and functionality that benefit the entire community.

What do you think of this idea?

Specifically…

  1. As an Add-on developer, do you like the idea of describing your project using a standard vocabulary and maintaining that alongside your source?
  2. As someone who might want to charge for Add-ons, would this save you time and energy, making it more likely that you would list (or even develop) your Add-on in the first place?
  3. As a customer, do you value having a one-stop shop for finding Alfresco extensions? Or is internet search good enough? If you found a promising Add-on that required an annual subscription, would you pay with a credit card if the service leveraged a trusted payment provider to handle your transaction safely and securely?

It’s a shame the old Add-ons site was demolished so hastily. The good news is we now have an opportunity for a do-over, hopefully without repeating past mistakes, putting the data back in the hands of the developers where it belongs, and restoring a community asset better-positioned for future growth.

 

Alfresco User Interface Options Revisited

Alfresco has been working hard on its new Application Development Framework (ADF), which consists of a client-side JavaScript API and a set of Angular components (see “Alfresco embraces Angular: Now what?“).

Alfresco says the ADF is now ready for production use (as of the 2.0 release) and they’ve also been iterating on an Example Content Application that shows how to use the ADF to build a general document management application.

Many of my clients are watching the ADF closely. Some are dabbling with Proofs-of-Concept, but few have started building their own ADF-based applications. One reason is purely practical–the ADF requires release 5.2 or higher, so an upgrade (or two) stands in the way of production ADF use.

The other reason is more strategic, and it centers around whether or not a customer should expect a content services platform vendor to provide a user interface, and, if so, what kind, or whether that should be the client’s responsibility. This is still being worked through, both internally and externally (see “Future of Alfresco Share Remains Foggy After DevCon“). Alfresco has been very open about their plans and has been gathering input from customers so we’ll see how it plays out.

Customers that require user interface customizations today may feel stuck–extensive Share customizations don’t make sense at this point, ADF requires 5.2, and, even if that’s not the problem, the amount of work required to assemble a solution using the framework and to maintain it going forward may not make sense, especially if what is needed is just vanilla document management with some UI tweaks. (I’m not saying “Don’t use ADF”–I’ve been lobbying for it since 2010. I’m just saying it might not make sense for everyone, for all use cases).

Nine years ago (yikes!) I wrote a post called “Alfresco User Interface: What Are My Options?“. It’s probably time to revise the list. If you need to provide custom functionality on top of Alfresco today, here are your options:

Use the ADF

If you are running 5.2 or higher, clearly, this is what Alfresco would recommend. It ships with components based on Angular, which I’ve been using for a while now and I really like. But, if Angular isn’t an option for you, you can leverage the client-side JavaScript library that is part of the ADF and use React or whatever you want.

And you don’t have to start with a blank app–you can use the Example Content Application as a starting point, if it’s close to what you need, or just as a learning tool if it isn’t.

Advantages:

  • Ready-to-use components save you a lot of work
  • Components and client-side JavaScript API supported by Alfresco
  • Angular is a popular framework, so there should be a lot of developers who can help
  • Client-side JavaScript API lets you use something other than Angular if needed
  • Heavy focus by Alfresco engineering means that new components and features are being added frequently

Considerations:

  • Requires Alfresco 5.2 or higher
  • While the components are extensible, you must stick to the extension points and not customize at the component source code level, unless you want to fork the framework (or at least that component) and maintain it going forward
  • Angular components are styled with Google Material Design which may or may not be aligned with your look-and-feel standards
  • If what you really need is Share with a certain set of customizations, it may be more work to re-build Share-like functionality using ADF components, at least in its current state

Use Your Own Framework

I’ve done several projects with custom, non-ADF Angular components hitting an API layer implemented with Spring Boot. The API layer talks to Alfresco via CMIS, the Alfresco REST API, custom web scripts, or some combination. The API layer also talks to other systems, which is important because these apps are rarely just about Alfresco alone.

I’ve been using Angular but you can obviously use whatever front-end you want. You don’t have to hit an API layer–you can hit Alfresco directly from your client-side JavaScript–the architecture, the tools, and the entire stack is up to you.

The nice thing about this approach is that it works with older versions of Alfresco. And the application only includes exactly what you need to meet end-user requirements. Of course, you have to build all of it yourself and maintain it going forward.

Advantages:

  • Total flexibility in the toolset and architecture
  • Meets your exact requirements

Considerations:

  • Custom code has to be written, debugged, and maintained
  • If you choose an esoteric or short-lived framework, you may be re-writing the application sooner than planned

Use a Commercial Front-end

I’ve seen some very slick commercial front-ends of late. If your main problem is presenting a compelling user interface for finding content, take a look at Alfred Finder from Xenit. It’s got some really impressive features for building and saving queries and a blazing fast user interface. Finder, as the name implies, is read-only. If you need to create content you’ll have to talk to Xenit about a customization or use something different.

For something more specific to case management, take a look at the OpenContent Management Suite from TSG. TSG has removed the hierarchical folder metaphor entirely. Instead, content just goes where it needs to go based on user role, the type of content, and the task at hand. The focus here is on end-user productivity where end-users are most likely case workers, records managers, or similar. (Despite TSG’s proclivity to use “Open” in its branding, this is not an open source solution. You must be a paying TSG consulting customer to use it and to see the source).

If your use case centers around forms, take a look at FormFactor. This isn’t a full replacement front-end like the previous two, but a lot of the customizations people do are really there to support custom data capture, so I’m including it because it might eliminate the need to do any customizations at all. FormFactor allows non-technical end-users to build and publish electronic forms via drag-and-drop, all within the existing Alfresco user interface. The demo I saw was built on top of Share. I’ve asked FormFactor via Twitter whether they will be able to support ADF-based clients as well but have not yet heard back.

Advantages:

  • Commercial add-ons offer shorter time-to-value
  • Maintained by a vendor
  • Functionality leverages the collective feedback of the vendor’s customers

Considerations:

  • May involve up-front license cost and/or annual maintenance fees
  • Commercial products are often shipped as-is, with a close, but not exact, fit to your requirements
  • Support SLA’s can differ widely from vendor-to-vendor
  • Generally speaking, working with your procurement department may not be considered one of the simple joys of life

Customizing Share is Not a Long-Term Option

Despite the announcement that parts of Share are being deprecated and the recommendation for all custom development to use ADF (announcement), I expect the Alfresco Share client to be around for quite a while. The transition to whatever comes after Share is likely to be lengthy and orderly. No timeframe has been announced, but my guess is this will be measured in years, not months.

So if you have a custom action here or there, or you want to remove a few features to streamline a bit, you should do so.  However, if you’ve got major Share renovations in mind, like stripping it down to the studs and knocking out a load-bearing wall or two, you are going to spend a small fortune on what will someday be throwaway work. Instead of doing that, think carefully about using one of the alternative approaches I’ve outlined in this post.

UPDATE: I changed all occurrences of “AngularJS” in this post to “Angular” to make it clear that what Alfresco uses (and what I’ve been using on my own projects) is the newer version of Angular that used to be known as “Angular2” but is now referred to just as “Angular”.

Photo Credit: wireframe, ipad, pencil & notes by Baldiri, CC-BY 2.0

Just for fun: Docker Swarm on my 4-node Raspberry Pi cluster

I recently spent some time standing up a four-node Raspberry Pi cluster running Docker and Docker Swarm. I had no real practical reason to do this–it just sounded fun. And it was!

Docker is a technology that allows you to package your applications together with the operating system into a virtual machine, called a container, that can run anywhere. Docker Swarm establishes a cluster of hosts which can be used to run one or more Docker-based containers. You tell Docker Swarm which containers you want to run and how many of each and it takes care of allocating those containers to machines, provisioning the containers, starting them up and keeping them running in case of failure.

For example, suppose you have an application that is comprised of a web server, an application server, a database, and a key-value store. Docker can be used to package up each of those tiers into containers. The web server container has a thin operating system, the web server, and your front-end code. The application server has a thin operating system, the application server, and your business logic. And so on.

That alone is useful. Containers can run anywhere–local developer machines, on-prem physical hardware, virtualized hardware, or in the cloud. Because the applications and the operating system they run on are packaged together as containers I don’t have to worry about installing and configuring the infrastructure plus the code every time a new instance is needed. Instead I just fire up the containers.

With Docker Swarm I can say, “Here is a fleet of servers. Here are my containers that make up my stack. Make sure I always have 6 web servers, 3 app servers, 2 databases, and 3 key-value stores running at all times. I don’t care which of the servers you use to do that, just make it happen.” And Docker Swarm takes care of it.

This works surprisingly well on Raspberry Pi. Sure, you could do it on beefier hardware, but it’s pretty fun to do it with machines no bigger than a pack of cards. I used a mix of Raspberry Pi models: 1 2b+ and 3 model 3b’s, but I’ve also seen it done with Pi Zero’s, which are even smaller.

The examples I’ll reference in the links below do simple things like install a node-based RESTful service that keeps track of a counter stored in Redis. But once you do that, it is easy to see how you could apply the same technique to other problems.

If you want to try it yourself, here are some resources that I found helpful:

If you don’t already have a multiple Raspberry Pi set up, here is a shopping list (with Amazon links):

I already had a 2b+ sitting around so I used that with 3 model 3’s. The performance difference between the 2b and the 3b was significant, though, so if I do much more with it I will replace the model 2 with another model 3. My existing model 2b+ has a Sense HAT attached to it, which, among other things, gives me a nice 8×8 RGB LED matrix for displaying messages and status indicators.

When it is all put together, it looks like this:

Last year I used my Raspberry Pi as part of a hands-on class I gave to some elementary school students for Hour of Code. I haven’t settled on what I might do for them this year or whether or not that will leverage my new cluster, but it is handy to have Docker running on my Pi’s because I can set stuff up, tear it down, and relocate it much more easily.

Have you tried the serverless framework?

Last year I was working on a POC. The target stack of the POC was to be 100% native AWS as much as possible. That’s when I came across Serverless. Back then it was still in beta, but I was really happy with it. After the POC was over I moved on to other things. A couple of days ago I was reminded how useful the framework is, so I thought I’d share some of those thoughts here.

Before I continue, a few words about the term, “serverless”. In short, it gets some folks riled up. I don’t want to debate whether or not it’s a useful term. What I like about the concept is that, as a developer, I can focus on my implementation details without worrying as much about the infrastructure the code is running on. In a “serverless” setup, my implementation is broken down into discrete functions that get instantiated and executed when invoked. Of course, there are servers somewhere, but I don’t have to give them a moment’s thought (nor do I have to pay to keep them running, at least not directly).

If your infrastructure provider of choice is AWS, functions run as part of a service offering called Lambda. If you want to expose those functions as RESTful endpoints, you can use the AWS API Gateway. Of course your Lambda functions can make calls to other AWS services such as Dynamo DB, S3, Simple Queue Service, and so on. For my POC, I leveraged all of those. And that’s where the serverless framework really comes in handy.

Anyone that has done anything with AWS knows it can often take a lot of clicks to get everything set up right. The serverless framework makes that easier by allowing me to declare my service, the functions that make up that service, and the resources those functions leverage, all in an easy-to-edit YAML file. Once you get that configuration done, you just tell serverless to deploy it, and it takes care of the rest.

Let’s say you want to create a simple service that returns some JSON. Serverless supports multiple languages including JavaScript, Python, and Java, but for now I’ll do a JavaScript example.

First, I’ll bootstrap the project:

serverless create --template aws-nodejs --path echo-service

The serverless framework creates a serverless.yml file and a sample function in handler.js that echoes back a lot of information about the request. It’s ready to deploy as-is. So, to test it out, I’ll deploy it with:

serverless deploy -v

Behind the scenes, the framework creates a cloud formation template and makes the AWS calls necessary to set everything up on the AWS side. This requires your AWS credentials to be configured, but that’s a one-time thing.

When the serverless framework is done deploying the service and its functions, I can invoke the sample function with:

serverless invoke -f hello -l

Which returns:

{
    "statusCode": 200,
    "body": "{\"message\":\"Go Serverless v1.0! Your function executed successfully!\",\"input\":{}}"
}

To invoke that function via a RESTful endpoint, I’ll edit serverless.yml file and add an HTTP event handler, like this:

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get

And then re-deploy:

serverless deploy -v

Now the function can be hit via curl:

curl https://someid999.execute-api.us-east-1.amazonaws.com/dev/hello

In this case, I showed an HTTP event triggering the function, but you can use other events to trigger functions, like when someone uploads something to S3, posts something to an SNS topic, or on a schedule. See the docs for a complete list.

To add additional functions, just edit handler.js and add a new function, then edit serverless.yml to update the list of functions.

Lambda functions cost nothing unless they are executed. AWS offers a generous free tier. Beyond the first million requests in a month it costs $0.20 per million requests (pricing).

I should also mention that if AWS is not your preferred provider, serverless also works with Azure, IBM, and Google.

Regardless of where you want to run it, if you’ve got 15 minutes you should definitely take a look at Serverless.

Tutorials updated for Alfresco SDK 3.0.0

A week or so ago Alfresco released version 3.0.0 of their Maven-based SDK. The release was pretty significant. There are a variety of places you can learn about the new SDK release:

I recently revised all of the Alfresco Developer Series tutorials to be up-to-date with SDK 3.

Upgrading each project was mostly smooth. The biggest change with SDK 3.0.0 is that the folder structure has changed slightly. At a high level, stuff that used to go in src/main/amp now goes in src/main/resources. These changes are good–they make the project structure look like any other Maven-based project which helps other developers (and tools) understand what’s going on.

I did have some problems getting the repo to build with an out-of-the-box project which I fixed by using the SNAPSHOT rather than the released build.

Support for integration tests seems to have changed as well. I actually still need to work that one out.

Other than that, it was mostly re-organizing the existing source and updating the pom.xml.

A small complaint is that the number of sample files provided with the SDK has grown significantly, which means the stuff you have to delete every time you create a new project has increased.

I understand that those are there for people just getting started, but no one, not even beginners, needs those files more than once, so I’d rather not see them in the main archetype. Maybe Alfresco should have a separate archetype called “sample project” that would include those, and configure the normal archetype as a completely empty project. Just a thought.

The new SDK is supposed to work with any release back to 4.2 so you should be able to upgrade to the new version for all of your projects.

This is How I Work

On any given day I might be writing code, designing the architecture for a content-centric solution, meeting with clients, collaborating with teammates, installing software on servers, writing blog posts, answering questions in forums, writing and signing contracts, or doing bookkeeping. Some days I do all of that. My day is probably similar to that of anyone else doing professional services work as a small business. Here are some of the tools I use to keep all of that going smoothly.

Hardware & Operating System

In 2006 I left Windows behind and never looked back. I ran Ubuntu as my primary desktop for three years, then switched to Mac OS X. I still love Linux, and all of my customers run Linux servers, but for my primary machine, I am most productive with my MacBook Pro and OS X.

I’ll admit that the latest MacBooks shook my faith by incorporating a shiny feature I’ll never use and not upgrading the CPU or RAM. I briefly considered moving back to Linux on something like a System76 or a Lenovo, but I am so deep into the Apple ecosystem in both home and office it may not be practical to switch. I’m hopeful the new MBP’s will get beefed up in terms of RAM and CPU later this year.

Collaborating with teammates and clients

I’ve been using Trello for project and task tracking for a long time and it really agrees with me. I set up Trello teams with some of my clients and it helps keep us all on track.

For real-time collaboration I tend to use Slack. It doesn’t always make sense to do so, but for selected projects, I invite my clients to my corporate Slack team and we use private channels to work on projects. We use Slack’s integrations to get notified when changes happen in Trello or in our codebase which resides in either Git or Bitbucket.

For real-time collaboration via IRC, Jabber, and GChat I use Adium.

Creating content & code

For plain text editing, I use either Aquamacs or Atom depending on what I’m doing. If it’s just a free-form text file, like maybe a blog post or just some rough meeting notes or something like that I’ll use Aquamacs, which is a Mac-specific distribution of Emacs. If I am doing non-compiled coding in something like JSON, XML, Python, Groovy, or JavaScript I’ll typically use Atom.

For more intense JavaScript projects I will often switch to WebStorm and sometimes for Python I’ll use PyCharm instead of Atom.

For Java projects I’ve recently moved from Eclipse to IntelliJ IDEA. I’ve used Eclipse for many, many years, but IntelliJ feels more reliable and polished. I’ve been pretty happy with it so far.

IntelliJ, WebStorm, and PyCharm are all available from JetBrains and the company offers an all-in-one subscription that is worth considering.

I do a lot with markdown. For example, if I’m taking notes on a customer’s installation and those notes will be shared with the customer, rather than just doing those notes in plain text with no structure, I’ll use markdown. Then, to preview the document and render it in PDF I use Marked. This is also handy for previewing Github readme files, but Atom can also be used for that.

Another time saver is using markdown to produce presentations. I wouldn’t necessarily use it for marketing-ready pitches, but for pulling together a quick deck to review thoughts with a customer or presenting a topic at a meetup, markdown is incredibly fast. To actually render and display the presentation from markdown I use Deckset. It produces beautiful presentations with very little effort while maintaining the editing speed that plain text markdown provides.

Sometimes I’ll create a video to illustrate a concept, either to help the community understand a feature or extension technique or to demo some new functionality to a customer when schedules won’t align. Telestream is a wonderful tool for creating such screencasts.

My open source projects live at Github while my closed source projects live at Bitbucket. The primary draw to Bitbucket is the free private repositories, but I really like what Atlassian offers. When git on the command-line just won’t do, I switch to Sourcetree, Atlassian’s visual git client.

Automation, Social & News Feeds

I have a client where I have to manage over 60 servers across several clusters. To automate the provisioning of new nodes, upgrades, and configuration management, I use Ansible. It’s much easier to learn than Chef and it requires no agents to be installed on the servers being managed. Plus, it’s Python-based.

Lots of servers, lots of customers, and lots of business and personal accounts means password management can become an issue. I use KeePassX on my desktop and iKeePass on my iOS devices to keep all of my credentials organized.

I have a ton of different news sources I try to keep up with. Feedly helps a lot. I use it on my mobile devices and on the web.

I have a personal Twitter account and a corporate Twitter account, plus I help out with other accounts from time-to-time. Luckily, Hootsuite makes that easy, and it handles more than just Twitter.

Back-office

When you run your own business there are two things that can be a time suck without the right tools: contracts/forms and bookkeeping. PDF Expert lets me fill in PDF forms and sign contracts and other documents right on my mobile device. It has direct integrations with Google Drive, Dropbox, and other file share services so it is easy to store signed documents wherever I need to.

I used QuickBooks Desktop Professional Services Edition to handle my bookkeeping for several years. But that edition was only for Windows, so I ran it in a Windows VM on my Mac with VMWare, which was kind of a drag. I finally got tired of that and migrated to QuickBooks Online. The migration was completely painless. I just called them up, and within about an hour they had taken my money and moved all of my data without anything getting screwed up. It was a pretty awesome experience.

Physical Office

I’ve been working from home for over ten years, so I appreciate the importance physical space plays in a productive working environment. My desk is a custom-built UPLIFT standing desk with a solid cherry top. I love it and haven’t had a problem with it. I do need to make myself stand up more often, though.

I found that even with the desk raised completely, my cinema display wasn’t quite high enough. So I snagged a Humanscale M8 articulating mount with an Apple VESA adapter. Now I just grab my display and put it exactly where it needs to be.

My Mac hooks up to my display via Thunderbolt, which makes connecting and disconnecting a breeze. But I didn’t like how much real estate the laptop took up on my desk. There are a variety of solutions for this. I went with a Twelve South BookArc vertical desktop stand and that’s worked really well. I have a minor concern about whether or not using a MacBook in a vertical position is bad with regard to heat dissipation but I’ve decided to roll the dice on that.

I love my home office setup, but I do get tired of the same four walls sometimes. To combat that and to just change things up a bit, every week I try to spend some time in a co-working space. Here in my hometown there’s one right on the square in the historic part of downtown that has got a good vibe and is close to good food. You might check Sharedesk to see if there’s something similar near you.

So there you have it. Those are some of the tools I use every day. Got any favorites you’d like to share?

The plain truth about Alfresco’s open source ethos

There was a small flare-up on the Order of the Bee list this week. It started when someone suggested that the Community Edition (CE) versus Enterprise Edition comparison page on alfresco.com put CE in a negative light. In full disclosure, I collaborated with Marketing on that page when I worked for Alfresco. My goal at the time was to make sure that the comparison was fair and that it didn’t disparage Community Edition. I think it still passes that test and is similar to the comparison pages of other commercial open source companies.

My response to the original post to the list was that people shouldn’t bother trying to get the page changed. Why? Because how Alfresco Software, Inc. chooses to market their software is out-of-scope for the community. As long as the commercial company behind Alfresco doesn’t say anything untrue about Community Edition, the community shouldn’t care.

The fact that there is a commercial company behind Alfresco, that they are in the business of selling Enterprise support subscriptions, and at the same time have a vested interest in promoting the use of Community Edition to certain market segments is something you have to get your head around.

Actually there are a handful of things that you really need to understand and accept so you can be a happy member of the community. Here they are:

1. CE is distributed under LGPLv3 so it is open source.

If you need to put a label on it and you are a binary type of person, this is at the top of the list. Alfresco is “open source” because it is distributed under an OSI-approved license. A more fine-grained description is that it is “open core” because the same software is distributed under two different licenses, with the enterprise version being based on the free version and including features not available in the free version.

2. Committers will only ever be employees.

There have been various efforts over the years to get the community more involved in making direct code contributions. The most recent is that Aikau is on github and accepting pull requests. Maybe some day the core repository will be donated to Apache or some other foundation. Until then, if you want to commit directly to core, send a resume to Alfresco Software, Inc. I know they are hiring talented engineers.

3. Alfresco Software, Inc. is a commercial, for-profit business.

Already mentioned, but worth repeating: The company behind the software earns revenue from support subscriptions, and, increasingly, value-added features not available in the open source distribution. The company is going to do everything it can to maximize revenue. The community needs this to be the case because a portion of those resources support the community product. The company needs the community, so it won’t do anything to aggressively undermine adoption of the free product. You have to believe this to be true. A certain amount of trust is required for a symbiotic relationship to work.

4. “Open source” is not a guiding principle for the company.

Individuals within the company are ardent open source advocates and passionate and valued community members, but the organization as a whole does not use “open source” as a fundamental guiding principle. This should not be surprising when you consider that:

  1. “Drive Open Innovation” not “Open Source” is a core value to the company as publicly expressed on the Our Values page.
  2. The leadership team has no open source experience (except John Newton and PHH whose open source experience is Alfresco and Activiti).
  3. The community team doesn’t exist any more–the company has shifted to a “developer engagement” strategy rather than having a dedicated community leadership or advocacy team.

Accept the fact that this is a software company like any other, that distributes some of its software under an open source license and employs many talented people who spend a lot of their time (on- and off-hours) to further the efforts of the community. It is not a “everything-we-do-we-do-because-open-source” kind of company. It just isn’t.

5. Alfresco originally released under an open source license primarily as a go-to-market strategy

In the early days, open source was attractive to the company not because it wanted help building the software, but because the license undermined the position of proprietary vendors and because they hoped to gain market share quickly by leveraging the viral nature of freely-distributable software. Being open was an attractive (and highly marketable) contrast to the extremely closed and proprietary nature of legacy ECM vendors such as EMC and Microsoft.

I think John and Paul also hoped that the open and transparent nature of open source would lend itself to developer adoption, third-party integrations and add-ons, and a partner ecosystem, which it did.

I think it is this last one–the mismatch between the original motivations to release as open source and what we as a community expect from an open source project–that causes angst. The “open source” moniker attracts people who wish the project was more like an organic open source project than it can or ever will be.

For me, personally, I accepted these as givens a long time ago–none of them bother me any more. I am taking this gift that we’ve been given–a highly-functional, freely-distributable ECM platform–and I’m using it to help people. I’m no longer interested in holding the company to a dogmatic standard they never intended to be held to.

So be cool and do your thing

The “commercial” part of “commercial open source” creates a tension that is felt both internally and externally. Internal tension happens when decisions have to be made for the benefit of one side at the expense of the other. External tension happens when the community feels like the company isn’t always acting in their best interest and lacks the context or visibility needed to believe otherwise.

This tension is a natural by-product of the commercial open source model. It will always be there. Let’s acknowledge it, but I see no reason to antagonize it.

If you want to help the community around Alfresco, participate. Build something. Install the software and help others get it up and running. Join the Order of the Bee. If you want to help Alfresco with its marketing, send them your resume.