Just published two minimal packages to help building micropub applications!
The first one is called micropub-parser and it's basically a port from @aaronpk's pk3-micropub package to JavaScript.
The second one is a small IndieAuth middleware that can be easily plugged into an Express.js app!
Testing out Malcolm Blaney's relatively new Unicyclic feed reader. Very clean and simple! I love that it handles Twitter feeds in a much more natural way than any of the big corporate feed readers out there. Of course the fact that it supports micropub is also a major bonus! IndieWeb tools FTW!
Syndicated copies to:
Made some significant progress today on an approximately 2,800 word feature article on the W3C Micropub spec with some motivation, examples, use cases, etc. Needs some editing, a bit of artwork, and some screenshots, but we’re nearly there.
Have a few ideas about placement already…
Syndicated copies to:Reading the #micropub spec. Seems simple enough, would anyone benefit from us adding that? Is anyone using that?
I’m always looking for fun and better ways to publish content to my website. There are several nice writing apps now, such as Byword for MacOS, which lets you write in Markdown and then converts it into HTML. Many of these kinds of apps have an option to publish to a Wordpress site, using Wordpress’ XML-RPC interface, which is more or less the MetaWeblog API.
I thought it would be a fun experiment to try to set up my website to handle those XML-RPC calls so that I can use these apps, at least until they support Micropub natively.
As with most XML formats, XML-RPC is ridiculously verbose for what it’s doing. In order to send a single string value, it requires wrapping it in a surprising amount of XML:
<param>
<value><string>aaronparecki.com</string></value>
</param>
The MetaWeblog API also requires that you give apps your Wordpress password, which has been a known anti-pattern for a long time. With things like OAuth, we have better ways of authenticating against APIs without sending passwords around.
To avoid needing to add actual XML-RPC support to my website, I set up a bridge that translates XML-RPC calls to their equivalent Micropub calls. The bridge also obtains an IndieAuth access token so that I can use a token instead of entering my password into these apps.
I didn’t bother adding any styling to the gateway since it’s something you’d only interact with once to set up, so apologies for the ugly screenshots that follow.
Here’s what it looks like to connect Byword to be able to publish to my website.
First I visit the gateway and log in.
That does the IndieAuth discovery and takes me to my website where I grant it access.
Then I’m redirected back to the gateway which provides instructions on what to do next.
The key to making Byword find the XML-RPC API is adding that EditURI tag to my website. Then I can go into Byword and add a new Wordpress site, entering my domain name.
It then asks for my username and password, which I enter by copying from the gateway.
Then Byword is all set and thinks it’s talking to a Wordpress site!
Now when I’m ready to publish this post, I click the “Publish” button in Byword, and fill in the title and tags.
If you’d like to try this out, head to xmlrpc.p3k.io and connect your website!
The source code is available on GitHub.
I’ve only implemented the one newPost method that Byword uses when talking to the XML-RPC API. I’ve only tested this with Byword, so it’s very likely that other apps might be expecting more of the API to be implemented. Feel free to file issues if you have trouble using other apps! Eventually I’d like to implement more of the MetaWeblog API in this gateway, even if I really don’t like writing XML-RPC code!
Over the last several months, I've been slowly putting the pieces in place to be able to build a solid indieweb reader. Today, I feel like I finally have enough in place to consider this functional enough that I am now using it every day!
One of the major missing pieces of the IndieWeb ecosystem has been having an integrated reading and posting experience that mirrors the ease with which it's possible to post and follow on Twitter and other silo apps.
We've seen a few attempts at indieweb readers over the past few years, but nothing has really taken off or stuck around. Even my own attempts at readers have fallen apart, both the previous iteration of Monocle in 2016, and my fork of selfoss in 2014. My suspicion has always been that we haven't seen many people building out this part of the ecosystem because it turns out there are a whole bunch of different parts to building a reader, many of which have no overlap in skillset: managing the subscription list, polling and fetching feeds, parsing feeds, data storage, rendering posts in a UI, providing inline action buttons to be able to reply and favorite posts, etc.
When I'm building out the UI components of a project, the last thing I want to have to think about is the invisible backend stuff like feed polling and parsing. Similarly when I'm tackling the problems with parsing and normalizing data from feeds, the last thing I am thinking about is Javascript button interactions. Not to mention that I am barely an iOS developer, so there's no way I'd be able to build out a full indieweb reader for iOS.
In April 2017, I started outlining a spec that draws a hard line between these very different parts of building a reader, and called it Microsub (for subscribing), as a complement to Micropub (for publishing). The basic idea is to separate the feed subscriptions from the UI parts of building a reader.
I started working from the ground up on building out the various aspects I knew I would need in order to eventually end up with a fully functional reader.
I based a lot of these design decisions around my previous experience in building a reader, as well as my documentation of how I use IRC to read content across the web.
My goal with this is to use this as my primary online dashboard to follow all kinds of content, as well as being able to interact with the content without leaving the interface.
The main organization of the reader is laid out in "channels". You can also think of these as "folders" if you want. Many feeds (or sources) can be added to a channel, and the posts are all combined into a single timeline view.
Monocle supports displaying a few different types of content in posts. It has native support for notes, articles, photos, multi-photos, videos, audio clips, checkins, favorites, reposts, and replies.
Since I follow my Instagram feed in the reader, I wanted to have a good display for photos, especially when there are many photos attached to a single post. I ended up doing a simple custom layout when there are two or three photos. Four or more they just start tiling as half-size photos. With three photos, the first photo appears larger on the left and the other two are stacked to the right.
I wanted to be able to read full articles in the reader without jumping out to the site, but also didn't want to have to scroll endlessly when I'm just skimming headlines. So if an article has content that is too tall, it gets truncated with a "read more" link to expand it.
If the post has an audio file, such as podcasts, then there is a simple HTML audio player inline!
Each post has a set of buttons to be able to respond to the post. The quickly accessible actions are "favorite", "repost", and "reply". The three dots opens up an expanded menu with some additional options, some of which I have not yet implemented.
Currently it's possible to remove a post from a channel, and to open up a debug view showing the raw data behind the post. Eventually this will expand to include muting the author, blocking the author, or unfollowing the source the post came from.
Clicking the "reply" button drops down a little text box for posting a reply.
There is a character counter just so that I have a sense of how long the post is. Since this is posting the reply back to my website, this interface has no idea what sort of character limits there are, so it's just a simple counter. When I click the "Reply" button, the app makes a Micropub post to my website to create the post there.
My website already has all the logic for adding that to my replies feed and sending webmentions to the post I'm replying to. Since this post I replied to is on Micro.blog, and Micro.blog accepts webmentions, the post showed up there within a few seconds.
The same workflow happens for favoriting and reposting things.
My website also recognizes when the post I'm replying to is a Twitter or GitHub permalink, and will automatically syndicate my reply or favorite appropriately! Since I added my GitHub notifications to a channel, I can actually reply to GitHub issues directly from the interface!
You may have noticed the little blue dots next to the channel names in some previous screenshots. Those indicate how many unread posts are in the channel.
However, some feeds that I follow end up with tons of posts in the channels, so many that the actual number of posts is no longer significant! All I really want to know is whether there is something new or not. To account for this, I can choose whether a channel implements per-item tracking, or just a boolean read/unread indicator, or disables read tracking altogether.
When I'm looking at a timeline, any new posts appear with a yellow glow around them. As the post scrolls out of view, it gets marked as read, and that state is pushed to the server so that other clients will also know it's now read.
I really enjoy not having to manually mark things as read, instead the interface just handles it all for me without any additional interaction.
Since I actually have several different websites I use, I wanted the response buttons to be able to post not just to my website, but also to the other websites I have. For example, my cat Dora, who has her own website is not always the best at using the computer, so sometimes I have to favorite things for her.
I can choose a alternate default account per channel so that the response buttons will actually post to the alternate website. Notice Dora's cat face in the bottom right corner of the screen. This lets me know that interacting with posts in this channel will be posted to this alternate account.
I can even temporarily switch to a different account by clicking on the profile icon and choosing another account.
You may also have noticed the little pen icon in the lower left corner. Clicking that pops up a dialog for writing a new post from the selected account. I chose to keep this interface super simple, providing just a text box and character counter.
If I need to write something more complicated, such as including HTML content, adding a photo, or choosing where the post is syndicated, then I'll just pop over to Quill and write the post there instead.
I mentioned earlier that there were many parts to this, and I haven't talked much about that yet. The most important thing about the architecture of this system is that it is not just a single monolithic app. Instead, there is a server responsible for collecting all the data from the feeds I'm following, and separate apps for displaying them! Since I was documenting everything on the wiki as I was building this out, other people were able to jump in and start writing clients from the beginning!
There are already two other great interfaces that work with the same backend server!
Here is a Javascript app called Together, written by Grant, showing the same posts you saw in a previous screenshot.
Here is the same content rendered by the iOS app, Indigenous, written by Eddie.
I'm pretty thrilled that already we've been able to have two people jump in and build readers so quickly already, thanks to the hard work of feed fetching being abstracted away by the server!
Now to start getting into the technical bits of how this works. Feel free to skip this section if specs make your eyes glaze over.
I mentioned before that the main separation going on here is splitting off the feed fetching and parsing from rendering the posts.
This accomplishes a few things:
The main idea behind this is the Microsub spec. This is the spec that the Microsub server implements so that clients know how they can talk to it.
Ideally there will eventually be a large ecosystem around the spec, with many clients to choose from, and many servers as well. We'll see some projects build in Microsub support natively, so that they work out of the box, and we'll also see some dedicated feed subscription services support Microsub. The nice thing about using your website identity to tie the pieces together is that you can choose your Microsub server separately from choosing the software that powers your website.
For example, I decided early on when building my website that I didn't want to mix the idea of following feeds into the same software that powers my website. So instead, I wrote an external Microsub server called Aperture, which is responsible for all the feed polling and parsing and storing the posts in channels. Aperture is open source, although I still consider it "in active development", so I am not officially supporting it right now. You are of course welcome to get it running yourself, but be prepared for things to change quickly.
(Aperture actually has two components, Watchtower which is a microservice that polls feeds and delivers them to Aperture itself, and Aperture does the actual feed parsing with the content provided by Watchtower. This allows me to scale out the feed polling separately from the Microsub server.)
If you want to try to get Aperture and Watchtower running, Daniel did a pretty great writeup of his experience getting things set up in two posts: Part 1 and Part 2.
Ideally I would love to see some more implementations of Microsub servers, so head over to the spec if that's your thing!
Monocle is the Microsub client I wrote that's featured in the screenshots above. It is also open source. Since it doesn't do any feed parsing itself, it doesn't even have a storage backend! Everything is fetched on the fly with the exception of the channel list and Micropub config which is persisted in the session data.
When I click on a channel to view it, Monocle first makes a Microsub timeline request to Aperture to fetch the channel data, then renders it in the timeline view. This is analogous to the iOS app Indigenous fetching the timeline data from the Microsub server then rendering it on the phone, except Monocle is doing that server-side to generate HTML for the browser.
You might think I'm crazy for having written a PHP app that fetches JSON from an API and then renders --gasp-- static HTML in 2018, but guess what -- it's fast!
Monocle is open source, but I am also hosting a version online that anyone is welcome to use at monocle.p3k.io. Since it doesn't actually store anything itself, I don't expect it to take up any significant resources any time soon! Of course in order to use it, you'll need to have your website pointing to a Microsub server of your choosing. Since that's where all the actual work is done, I am not making my hosted version of Aperture available for general signups right now. You'll need to either get that running on your own server, or build a Microsub server from scratch!
This last section has been a bit of a wall of text, so here is a diagram showing how all the pieces fit together to make this possible!
My website contains the IndieAuth and Micropub bits, but others have chosen to use external services for those as well. I've also chosen to outsource sending and receiving webmentions to external services, whereas other people end up handling those within their own website code as well.
The reader apps all talk to the Microsub server that I've linked to in order to view posts, and when I tap a "like" button or write a reply from the app, they post that to my Micropub server to create a new post.
I use webmention.io to handle my incoming webmentions, and it is configured to send posts to a channel in Aperture using Aperture's Micropub API.
This has been a very long read, so congrats if you've made it this far! Here are some links if you're curious about how you can start building out various pieces of the ecosystem as well!
Specs
Open Source Projects
While this is all a good start, and I do actually use this as my primary online home now, there is still a lot more work to do!
As always, I'm happy to chat about any and all of this! It's been a lot of fun already to build this all out and see it working! You can find me in the IndieWeb chat via IRC and Slack, if you send me a Webmention I'll see it in my reader, or find me at an upcoming IndieWeb event!
Just implemented mp-destination on my site! I can connect multiple Micropub or Twitter accounts to my site to be able to post to other accounts from Micropub clients when I'm logged in as me. This is the first step of turning Monocle into more of a Tweetdeck-like experience!
EdwardHinkle
https://github.com/EdwardHinkle
•
Feb 20
Yep, exactly. Or a specialized app like Teacup that sends ate and drank posts. Those should still go through regardless.
As you said, @manton, it's more of a suggestion but especially a suggestion for generalized Micropub application, as opposed to specialized.
manton
https://github.com/manton
•
Feb 20
@aaronpk I'm glad you mentioned Post Type Discovery, because to me that is the part to focus on. It seems too complicated to require spelling out every property, like category or photo. If an endpoint doesn't support accepting a category on a new post, no harm done.
What can we borrow from the Post Type Discovery spec that will help here? At the very least it seems like the Microformats class names should be consistent.
In my example (https://indieweb.org/Micropub-brainstorming#Query_for_supported_vocabulary) I included what I view as the common actions from an app like Indigenous: like-of, repost-of, and bookmark-of, but bookmarks aren't actually mentioned in Post Type Discovery. I wonder if they should be, or are they not different enough from a regular post to list separately?
Interesting, I didn't actually realize bookmark wasn't in Post Type Discovery. It looks like it was mentioned under "Other Types Under Consideration" before it was moved to the W3C repo. Now the W3C note links to the Kinds of Posts section on the IndieWeb wiki for that.
The way we were adding things to the list of types in the algorithm was roughly based on how well-established the markup was in the wild. I am kind of surprised bookmarks didn't make that cut, but oh well.
The one potential confusion here is that post types are not the same as h-* types, e.g. there is no h-reply because you use the in-reply-to property on h-entry to create a reply post. I think that just means we need to be explicit about what to call this. To build on your previous example, this could be a solution:
{
"post-types": [
{
"type": "note",
"name": "Note"
},
{
"type": "article",
"name": "Blog Post"
},
{
"type": "photo",
"name": "Photo"
},
{
"type": "video",
"name": "Video"
},
{
"type": "reply",
"name": "Reply"
},
{
"type": "like",
"name": "Like"
},
{
"type": "repost",
"name": "Repost"
},
{
"type": "rsvp",
"name": "RSVP"
},
{
"type": "bookmark",
"name": "Bookmark"
}
]
}
Clients should assume that if it's not in the list, then the server doesn't support it? Of course there needs to be some sensible behavior for servers that don't return this info at all.
Would it make sense to omit note from this list since that's kind of a baseline? Or keep it in the list because it allows the client to customize the name of the button still?
manton
https://github.com/manton
•
Feb 20
@EdwardHinkle Sounds great! I can implement it at any time on Micro.blog, although first I wonder if anyone has feedback on the JSON key names.
One thing to note is these are all properties of the h-entry vocabulary, whereas other kinds of posts support totally different properties. Things like h-event or h-review where they are completely different things. Continuing down this path it would make sense to scope these properties to note that they are part of the h-entry vocabulary.
The other question is how many of the supported properties would need to be defined? If not all of them, (e.g. photo, category, published) why these ones in particular?
Is it because these correspond with post types? If that's the case, then maybe this should be somehow tied to the Post Type Discovery spec which spells out what properties map to what kinds of posts.
In that case, it may make more sense to have the server advertise which of these post types it supports, and then clients would look at the spec to know which properties to send to create those posts.