Over the last several months, I've been slowly putting the pieces in place to be able to build a solid indieweb reader. Today, I feel like I finally have enough in place to consider this functional enough that I am now using it every day!
One of the major missing pieces of the IndieWeb ecosystem has been having an integrated reading and posting experience that mirrors the ease with which it's possible to post and follow on Twitter and other silo apps.
We've seen a few attempts at indieweb readers over the past few years, but nothing has really taken off or stuck around. Even my own attempts at readers have fallen apart, both the previous iteration of Monocle in 2016, and my fork of selfoss in 2014. My suspicion has always been that we haven't seen many people building out this part of the ecosystem because it turns out there are a whole bunch of different parts to building a reader, many of which have no overlap in skillset: managing the subscription list, polling and fetching feeds, parsing feeds, data storage, rendering posts in a UI, providing inline action buttons to be able to reply and favorite posts, etc.
When I'm building out the UI components of a project, the last thing I want to have to think about is the invisible backend stuff like feed polling and parsing. Similarly when I'm tackling the problems with parsing and normalizing data from feeds, the last thing I am thinking about is Javascript button interactions. Not to mention that I am barely an iOS developer, so there's no way I'd be able to build out a full indieweb reader for iOS.
In April 2017, I started outlining a spec that draws a hard line between these very different parts of building a reader, and called it Microsub (for subscribing), as a complement to Micropub (for publishing). The basic idea is to separate the feed subscriptions from the UI parts of building a reader.
I started working from the ground up on building out the various aspects I knew I would need in order to eventually end up with a fully functional reader.
I based a lot of these design decisions around my previous experience in building a reader, as well as my documentation of how I use IRC to read content across the web.
My goal with this is to use this as my primary online dashboard to follow all kinds of content, as well as being able to interact with the content without leaving the interface.
Channels
The main organization of the reader is laid out in "channels". You can also think of these as "folders" if you want. Many feeds (or sources) can be added to a channel, and the posts are all combined into a single timeline view.
Displaying Posts
Monocle supports displaying a few different types of content in posts. It has native support for notes, articles, photos, multi-photos, videos, audio clips, checkins, favorites, reposts, and replies.
Since I follow my Instagram feed in the reader, I wanted to have a good display for photos, especially when there are many photos attached to a single post. I ended up doing a simple custom layout when there are two or three photos. Four or more they just start tiling as half-size photos. With three photos, the first photo appears larger on the left and the other two are stacked to the right.
I wanted to be able to read full articles in the reader without jumping out to the site, but also didn't want to have to scroll endlessly when I'm just skimming headlines. So if an article has content that is too tall, it gets truncated with a "read more" link to expand it.
If the post has an audio file, such as podcasts, then there is a simple HTML audio player inline!
Post Actions
Each post has a set of buttons to be able to respond to the post. The quickly accessible actions are "favorite", "repost", and "reply". The three dots opens up an expanded menu with some additional options, some of which I have not yet implemented.
Currently it's possible to remove a post from a channel, and to open up a debug view showing the raw data behind the post. Eventually this will expand to include muting the author, blocking the author, or unfollowing the source the post came from.
Replying
Clicking the "reply" button drops down a little text box for posting a reply.
There is a character counter just so that I have a sense of how long the post is. Since this is posting the reply back to my website, this interface has no idea what sort of character limits there are, so it's just a simple counter. When I click the "Reply" button, the app makes a Micropub post to my website to create the post there.
My website already has all the logic for adding that to my replies feed and sending webmentions to the post I'm replying to. Since this post I replied to is on Micro.blog, and Micro.blog accepts webmentions, the post showed up there within a few seconds.
The same workflow happens for favoriting and reposting things.
Syndicating to GitHub
My website also recognizes when the post I'm replying to is a Twitter or GitHub permalink, and will automatically syndicate my reply or favorite appropriately! Since I added my GitHub notifications to a channel, I can actually reply to GitHub issues directly from the interface!
Read-State Tracking
You may have noticed the little blue dots next to the channel names in some previous screenshots. Those indicate how many unread posts are in the channel.
However, some feeds that I follow end up with tons of posts in the channels, so many that the actual number of posts is no longer significant! All I really want to know is whether there is something new or not. To account for this, I can choose whether a channel implements per-item tracking, or just a boolean read/unread indicator, or disables read tracking altogether.
When I'm looking at a timeline, any new posts appear with a yellow glow around them. As the post scrolls out of view, it gets marked as read, and that state is pushed to the server so that other clients will also know it's now read.
I really enjoy not having to manually mark things as read, instead the interface just handles it all for me without any additional interaction.
Multiple Website Support
Since I actually have several different websites I use, I wanted the response buttons to be able to post not just to my website, but also to the other websites I have. For example, my cat Dora, who has her own website is not always the best at using the computer, so sometimes I have to favorite things for her.
I can choose a alternate default account per channel so that the response buttons will actually post to the alternate website. Notice Dora's cat face in the bottom right corner of the screen. This lets me know that interacting with posts in this channel will be posted to this alternate account.
I can even temporarily switch to a different account by clicking on the profile icon and choosing another account.
Simple Posting Interface
You may also have noticed the little pen icon in the lower left corner. Clicking that pops up a dialog for writing a new post from the selected account. I chose to keep this interface super simple, providing just a text box and character counter.
If I need to write something more complicated, such as including HTML content, adding a photo, or choosing where the post is syndicated, then I'll just pop over to Quill and write the post there instead.
Multiple Apps
I mentioned earlier that there were many parts to this, and I haven't talked much about that yet. The most important thing about the architecture of this system is that it is not just a single monolithic app. Instead, there is a server responsible for collecting all the data from the feeds I'm following, and separate apps for displaying them! Since I was documenting everything on the wiki as I was building this out, other people were able to jump in and start writing clients from the beginning!
There are already two other great interfaces that work with the same backend server!
Here is a Javascript app called Together, written by Grant, showing the same posts you saw in a previous screenshot.
Here is the same content rendered by the iOS app, Indigenous, written by Eddie.
I'm pretty thrilled that already we've been able to have two people jump in and build readers so quickly already, thanks to the hard work of feed fetching being abstracted away by the server!
The Microsub Server
Now to start getting into the technical bits of how this works. Feel free to skip this section if specs make your eyes glaze over.
I mentioned before that the main separation going on here is splitting off the feed fetching and parsing from rendering the posts.
This accomplishes a few things:
- Enables app developers to focus on the UI aspects of building a reader
- Allows you to choose which service you want to use to manage your subscriptions
- Enables you to use many different reader apps all talking to the same server backend that you control
- Leaves room for servers to do experimental things with feeds and subscriptions (think "magic" or "smart" feeds) without having to bother with the UI components or needing to get clients to add support
The main idea behind this is the Microsub spec. This is the spec that the Microsub server implements so that clients know how they can talk to it.
Ideally there will eventually be a large ecosystem around the spec, with many clients to choose from, and many servers as well. We'll see some projects build in Microsub support natively, so that they work out of the box, and we'll also see some dedicated feed subscription services support Microsub. The nice thing about using your website identity to tie the pieces together is that you can choose your Microsub server separately from choosing the software that powers your website.
For example, I decided early on when building my website that I didn't want to mix the idea of following feeds into the same software that powers my website. So instead, I wrote an external Microsub server called Aperture, which is responsible for all the feed polling and parsing and storing the posts in channels. Aperture is open source, although I still consider it "in active development", so I am not officially supporting it right now. You are of course welcome to get it running yourself, but be prepared for things to change quickly.
(Aperture actually has two components, Watchtower which is a microservice that polls feeds and delivers them to Aperture itself, and Aperture does the actual feed parsing with the content provided by Watchtower. This allows me to scale out the feed polling separately from the Microsub server.)
If you want to try to get Aperture and Watchtower running, Daniel did a pretty great writeup of his experience getting things set up in two posts: Part 1 and Part 2.
Ideally I would love to see some more implementations of Microsub servers, so head over to the spec if that's your thing!
Monocle
Monocle is the Microsub client I wrote that's featured in the screenshots above. It is also open source. Since it doesn't do any feed parsing itself, it doesn't even have a storage backend! Everything is fetched on the fly with the exception of the channel list and Micropub config which is persisted in the session data.
When I click on a channel to view it, Monocle first makes a Microsub timeline request to Aperture to fetch the channel data, then renders it in the timeline view. This is analogous to the iOS app Indigenous fetching the timeline data from the Microsub server then rendering it on the phone, except Monocle is doing that server-side to generate HTML for the browser.
You might think I'm crazy for having written a PHP app that fetches JSON from an API and then renders --gasp-- static HTML in 2018, but guess what -- it's fast!
Monocle is open source, but I am also hosting a version online that anyone is welcome to use at monocle.p3k.io. Since it doesn't actually store anything itself, I don't expect it to take up any significant resources any time soon! Of course in order to use it, you'll need to have your website pointing to a Microsub server of your choosing. Since that's where all the actual work is done, I am not making my hosted version of Aperture available for general signups right now. You'll need to either get that running on your own server, or build a Microsub server from scratch!
Putting the Pieces Together
This last section has been a bit of a wall of text, so here is a diagram showing how all the pieces fit together to make this possible!
My website contains the IndieAuth and Micropub bits, but others have chosen to use external services for those as well. I've also chosen to outsource sending and receiving webmentions to external services, whereas other people end up handling those within their own website code as well.
The reader apps all talk to the Microsub server that I've linked to in order to view posts, and when I tap a "like" button or write a reply from the app, they post that to my Micropub server to create a new post.
I use webmention.io to handle my incoming webmentions, and it is configured to send posts to a channel in Aperture using Aperture's Micropub API.
Further Reading
This has been a very long read, so congrats if you've made it this far! Here are some links if you're curious about how you can start building out various pieces of the ecosystem as well!
Specs
-
Microsub - a standardized way for apps to consume and interact with feeds collected by a server
- Micropub - a W3C Recommendation for creating, updating and deleting posts using external apps
- Webmention - a W3C Recommendation enabling cross-site comments and other interactions
- IndieAuth - an OAuth 2.0 extension that enables you to authorize third-party apps to talk to your Micropub or Microsub servers
- mp-destination - a Micropub extension allowing a server to designate alternate destinations for creating posts
Open Source Projects
-
Indigenous - an iOS Microsub client
- Together - a React JS Microsub client
- Monocle - a server-side PHP Microsub client
- Aperture - a PHP Microsub server
- Watchtower - a feed fetching microservice
Future Work
While this is all a good start, and I do actually use this as my primary online home now, there is still a lot more work to do!
-
More Microsub servers! I want to see at least two more solid Microsub server implementations in the relatively near future! That will help develop the spec further and ensure we're actually building interoperable tools. I suspect one of the implementations will end up being part of an integrated CMS such as Known or Wordpress, or will be a proxy to an existing feed reader service.
- Following private content. Private content has always been a challenge, mainly due to the fact that any time authentication is involved it complicates things a lot. With IndieAuth finally written up as a spec, we now have a solid building block to use to experiment in this area more.
- An Android Microsub app. While both Monocle and Together work pretty well on mobile browsers, there are still many advantages to having a native Android app! And it sounds like one is already in the works.
- Better UI for actually following people. I've kind of taken a shortcut on this front in order to move the rest forward. Right now, you still need to type someone's URL into a Microsub app in order to follow them. There are many challenges with streamlining this process further.
As always, I'm happy to chat about any and all of this! It's been a lot of fun already to build this all out and see it working! You can find me in the IndieWeb chat via IRC and Slack, if you send me a Webmention I'll see it in my reader, or find me at an upcoming IndieWeb event!
RSVP http://tantek.com/2018/080/e1
Special guest @aaronpk will demo his #IndieWeb reader setup!
https://aaronparecki.com/2018/03/12/17/building-an-indieweb-reader built on #openweb standards #WebSub #Microsub #microformats2 #IndieAuth #MicroPub #Webmention #Webhooks