Reading the #micropub spec. Seems simple enough, would anyone benefit from us adding that? Is anyone using that?
#MicroPub
-
My micropub endpoint is now hit when I “like” an article in instapaper. I have an applet setup on IFTTT that uses their webhook action. Exciting! Next up, auto posting instapaper highlights. #indieweb #micropub https://mcmanus.io/notes/2018/09/11/my-micropub-endpoint-is-now-hit-when-i-like-an-article-in-instapaper
-
@danielpunkass @manton It's been a long-time coming and we're still experimenting with the post list query for Micropub, but it'd be great if you guys could take a glance at the current discussions and let us know if there is anything missing.
-
A MetaWeblog to Micropub Gateway
I’m always looking for fun and better ways to publish content to my website. There are several nice writing apps now, such as Byword for MacOS, which lets you write in Markdown and then converts it into HTML. Many of these kinds of apps have an option to publish to a Wordpress site, using Wordpress’ XML-RPC interface, which is more or less the MetaWeblog API.
I thought it would be a fun experiment to try to set up my website to handle those XML-RPC calls so that I can use these apps, at least until they support Micropub natively.
Problems with XML-RPC
As with most XML formats, XML-RPC is ridiculously verbose for what it’s doing. In order to send a single string value, it requires wrapping it in a surprising amount of XML:
<param> <value><string>aaronparecki.com</string></value> </param>The MetaWeblog API also requires that you give apps your Wordpress password, which has been a known anti-pattern for a long time. With things like OAuth, we have better ways of authenticating against APIs without sending passwords around.
Micropub Bridge
To avoid needing to add actual XML-RPC support to my website, I set up a bridge that translates XML-RPC calls to their equivalent Micropub calls. The bridge also obtains an IndieAuth access token so that I can use a token instead of entering my password into these apps.
I didn’t bother adding any styling to the gateway since it’s something you’d only interact with once to set up, so apologies for the ugly screenshots that follow.
Here’s what it looks like to connect Byword to be able to publish to my website.
First I visit the gateway and log in.
That does the IndieAuth discovery and takes me to my website where I grant it access.
Then I’m redirected back to the gateway which provides instructions on what to do next.
The key to making Byword find the XML-RPC API is adding that
EditURItag to my website. Then I can go into Byword and add a new Wordpress site, entering my domain name.
It then asks for my username and password, which I enter by copying from the gateway.
Then Byword is all set and thinks it’s talking to a Wordpress site!
Now when I’m ready to publish this post, I click the “Publish” button in Byword, and fill in the title and tags.
Open Source
If you’d like to try this out, head to xmlrpc.p3k.io and connect your website!
The source code is available on GitHub.
I’ve only implemented the one
newPostmethod that Byword uses when talking to the XML-RPC API. I’ve only tested this with Byword, so it’s very likely that other apps might be expecting more of the API to be implemented. Feel free to file issues if you have trouble using other apps! Eventually I’d like to implement more of the MetaWeblog API in this gateway, even if I really don’t like writing XML-RPC code! -
I'm really on the fence about using HTTP 5xx responses to indicate any sort of application-level information. Currently any time a client gets an HTTP 5xx error, it can assume something went wrong with the *transport* of the request, such as a gateway timeout.
In REST APIs, HTTP 501 indicates that the HTTP method itself is unsupported, which in this case is a POST request. Micropub is not a REST API though, so I would be worried that a 501 could inadvertently indicate to a client that a POST request is not supported.
Can you find examples of other non-REST APIs that use HTTP 501 for this feature? I wouldn't want to recommend 501 unless there are existing examples of other non-REST or RPC APIs using it. -
hosting Homebrew Website Club SF tonight @MozSF!
RSVP http://tantek.com/2018/080/e1
Special guest @aaronpk will demo his #IndieWeb reader setup!
https://aaronparecki.com/2018/03/12/17/building-an-indieweb-reader built on #openweb standards #WebSub #Microsub #microformats2 #IndieAuth #MicroPub #Webmention #Webhooks -
Building an IndieWeb Reader
Over the last several months, I've been slowly putting the pieces in place to be able to build a solid indieweb reader. Today, I feel like I finally have enough in place to consider this functional enough that I am now using it every day!
One of the major missing pieces of the IndieWeb ecosystem has been having an integrated reading and posting experience that mirrors the ease with which it's possible to post and follow on Twitter and other silo apps.
We've seen a few attempts at indieweb readers over the past few years, but nothing has really taken off or stuck around. Even my own attempts at readers have fallen apart, both the previous iteration of Monocle in 2016, and my fork of selfoss in 2014. My suspicion has always been that we haven't seen many people building out this part of the ecosystem because it turns out there are a whole bunch of different parts to building a reader, many of which have no overlap in skillset: managing the subscription list, polling and fetching feeds, parsing feeds, data storage, rendering posts in a UI, providing inline action buttons to be able to reply and favorite posts, etc.
When I'm building out the UI components of a project, the last thing I want to have to think about is the invisible backend stuff like feed polling and parsing. Similarly when I'm tackling the problems with parsing and normalizing data from feeds, the last thing I am thinking about is Javascript button interactions. Not to mention that I am barely an iOS developer, so there's no way I'd be able to build out a full indieweb reader for iOS.
In April 2017, I started outlining a spec that draws a hard line between these very different parts of building a reader, and called it Microsub (for subscribing), as a complement to Micropub (for publishing). The basic idea is to separate the feed subscriptions from the UI parts of building a reader.
I started working from the ground up on building out the various aspects I knew I would need in order to eventually end up with a fully functional reader.
The main interface of Monocle I based a lot of these design decisions around my previous experience in building a reader, as well as my documentation of how I use IRC to read content across the web.
My goal with this is to use this as my primary online dashboard to follow all kinds of content, as well as being able to interact with the content without leaving the interface.
Channels
The main organization of the reader is laid out in "channels". You can also think of these as "folders" if you want. Many feeds (or sources) can be added to a channel, and the posts are all combined into a single timeline view.
Displaying Posts
Monocle supports displaying a few different types of content in posts. It has native support for notes, articles, photos, multi-photos, videos, audio clips, checkins, favorites, reposts, and replies.
Since I follow my Instagram feed in the reader, I wanted to have a good display for photos, especially when there are many photos attached to a single post. I ended up doing a simple custom layout when there are two or three photos. Four or more they just start tiling as half-size photos. With three photos, the first photo appears larger on the left and the other two are stacked to the right.
I wanted to be able to read full articles in the reader without jumping out to the site, but also didn't want to have to scroll endlessly when I'm just skimming headlines. So if an article has content that is too tall, it gets truncated with a "read more" link to expand it.
If the post has an audio file, such as podcasts, then there is a simple HTML audio player inline!
Post Actions
Each post has a set of buttons to be able to respond to the post. The quickly accessible actions are "favorite", "repost", and "reply". The three dots opens up an expanded menu with some additional options, some of which I have not yet implemented.
Currently it's possible to remove a post from a channel, and to open up a debug view showing the raw data behind the post. Eventually this will expand to include muting the author, blocking the author, or unfollowing the source the post came from.
Replying
Clicking the "reply" button drops down a little text box for posting a reply.
There is a character counter just so that I have a sense of how long the post is. Since this is posting the reply back to my website, this interface has no idea what sort of character limits there are, so it's just a simple counter. When I click the "Reply" button, the app makes a Micropub post to my website to create the post there.
My website already has all the logic for adding that to my replies feed and sending webmentions to the post I'm replying to. Since this post I replied to is on Micro.blog, and Micro.blog accepts webmentions, the post showed up there within a few seconds.
The same workflow happens for favoriting and reposting things.
Syndicating to GitHub
My website also recognizes when the post I'm replying to is a Twitter or GitHub permalink, and will automatically syndicate my reply or favorite appropriately! Since I added my GitHub notifications to a channel, I can actually reply to GitHub issues directly from the interface!
Replying to a GitHub issue comment from Monocle 
My reply to the GitHub issue on my website 
My reply automatically syndicated to GitHub Read-State Tracking
You may have noticed the little blue dots next to the channel names in some previous screenshots. Those indicate how many unread posts are in the channel.
However, some feeds that I follow end up with tons of posts in the channels, so many that the actual number of posts is no longer significant! All I really want to know is whether there is something new or not. To account for this, I can choose whether a channel implements per-item tracking, or just a boolean read/unread indicator, or disables read tracking altogether.
When I'm looking at a timeline, any new posts appear with a yellow glow around them. As the post scrolls out of view, it gets marked as read, and that state is pushed to the server so that other clients will also know it's now read.
I really enjoy not having to manually mark things as read, instead the interface just handles it all for me without any additional interaction.
Multiple Website Support
Since I actually have several different websites I use, I wanted the response buttons to be able to post not just to my website, but also to the other websites I have. For example, my cat Dora, who has her own website is not always the best at using the computer, so sometimes I have to favorite things for her.
I can choose a alternate default account per channel so that the response buttons will actually post to the alternate website. Notice Dora's cat face in the bottom right corner of the screen. This lets me know that interacting with posts in this channel will be posted to this alternate account.
I can even temporarily switch to a different account by clicking on the profile icon and choosing another account.
Simple Posting Interface
You may also have noticed the little pen icon in the lower left corner. Clicking that pops up a dialog for writing a new post from the selected account. I chose to keep this interface super simple, providing just a text box and character counter.
If I need to write something more complicated, such as including HTML content, adding a photo, or choosing where the post is syndicated, then I'll just pop over to Quill and write the post there instead.
Multiple Apps
I mentioned earlier that there were many parts to this, and I haven't talked much about that yet. The most important thing about the architecture of this system is that it is not just a single monolithic app. Instead, there is a server responsible for collecting all the data from the feeds I'm following, and separate apps for displaying them! Since I was documenting everything on the wiki as I was building this out, other people were able to jump in and start writing clients from the beginning!
There are already two other great interfaces that work with the same backend server!
Here is a Javascript app called Together, written by Grant, showing the same posts you saw in a previous screenshot.
Here is the same content rendered by the iOS app, Indigenous, written by Eddie.
I'm pretty thrilled that already we've been able to have two people jump in and build readers so quickly already, thanks to the hard work of feed fetching being abstracted away by the server!
The Microsub Server
Now to start getting into the technical bits of how this works. Feel free to skip this section if specs make your eyes glaze over.
I mentioned before that the main separation going on here is splitting off the feed fetching and parsing from rendering the posts.
This accomplishes a few things:
- Enables app developers to focus on the UI aspects of building a reader
- Allows you to choose which service you want to use to manage your subscriptions
- Enables you to use many different reader apps all talking to the same server backend that you control
- Leaves room for servers to do experimental things with feeds and subscriptions (think "magic" or "smart" feeds) without having to bother with the UI components or needing to get clients to add support
The main idea behind this is the Microsub spec. This is the spec that the Microsub server implements so that clients know how they can talk to it.
Ideally there will eventually be a large ecosystem around the spec, with many clients to choose from, and many servers as well. We'll see some projects build in Microsub support natively, so that they work out of the box, and we'll also see some dedicated feed subscription services support Microsub. The nice thing about using your website identity to tie the pieces together is that you can choose your Microsub server separately from choosing the software that powers your website.
For example, I decided early on when building my website that I didn't want to mix the idea of following feeds into the same software that powers my website. So instead, I wrote an external Microsub server called Aperture, which is responsible for all the feed polling and parsing and storing the posts in channels. Aperture is open source, although I still consider it "in active development", so I am not officially supporting it right now. You are of course welcome to get it running yourself, but be prepared for things to change quickly.
(Aperture actually has two components, Watchtower which is a microservice that polls feeds and delivers them to Aperture itself, and Aperture does the actual feed parsing with the content provided by Watchtower. This allows me to scale out the feed polling separately from the Microsub server.)
If you want to try to get Aperture and Watchtower running, Daniel did a pretty great writeup of his experience getting things set up in two posts: Part 1 and Part 2.
Ideally I would love to see some more implementations of Microsub servers, so head over to the spec if that's your thing!
Monocle
Monocle is the Microsub client I wrote that's featured in the screenshots above. It is also open source. Since it doesn't do any feed parsing itself, it doesn't even have a storage backend! Everything is fetched on the fly with the exception of the channel list and Micropub config which is persisted in the session data.
When I click on a channel to view it, Monocle first makes a Microsub timeline request to Aperture to fetch the channel data, then renders it in the timeline view. This is analogous to the iOS app Indigenous fetching the timeline data from the Microsub server then rendering it on the phone, except Monocle is doing that server-side to generate HTML for the browser.
You might think I'm crazy for having written a PHP app that fetches JSON from an API and then renders --gasp-- static HTML in 2018, but guess what -- it's fast!
Monocle is open source, but I am also hosting a version online that anyone is welcome to use at monocle.p3k.io. Since it doesn't actually store anything itself, I don't expect it to take up any significant resources any time soon! Of course in order to use it, you'll need to have your website pointing to a Microsub server of your choosing. Since that's where all the actual work is done, I am not making my hosted version of Aperture available for general signups right now. You'll need to either get that running on your own server, or build a Microsub server from scratch!
Putting the Pieces Together
This last section has been a bit of a wall of text, so here is a diagram showing how all the pieces fit together to make this possible!
My website contains the IndieAuth and Micropub bits, but others have chosen to use external services for those as well. I've also chosen to outsource sending and receiving webmentions to external services, whereas other people end up handling those within their own website code as well.
The reader apps all talk to the Microsub server that I've linked to in order to view posts, and when I tap a "like" button or write a reply from the app, they post that to my Micropub server to create a new post.
I use webmention.io to handle my incoming webmentions, and it is configured to send posts to a channel in Aperture using Aperture's Micropub API.
Further Reading
This has been a very long read, so congrats if you've made it this far! Here are some links if you're curious about how you can start building out various pieces of the ecosystem as well!
Specs
-
Microsub - a standardized way for apps to consume and interact with feeds collected by a server
- Micropub - a W3C Recommendation for creating, updating and deleting posts using external apps
- Webmention - a W3C Recommendation enabling cross-site comments and other interactions
- IndieAuth - an OAuth 2.0 extension that enables you to authorize third-party apps to talk to your Micropub or Microsub servers
- mp-destination - a Micropub extension allowing a server to designate alternate destinations for creating posts
Open Source Projects
-
Indigenous - an iOS Microsub client
- Together - a React JS Microsub client
- Monocle - a server-side PHP Microsub client
- Aperture - a PHP Microsub server
- Watchtower - a feed fetching microservice
Future Work
While this is all a good start, and I do actually use this as my primary online home now, there is still a lot more work to do!
-
More Microsub servers! I want to see at least two more solid Microsub server implementations in the relatively near future! That will help develop the spec further and ensure we're actually building interoperable tools. I suspect one of the implementations will end up being part of an integrated CMS such as Known or Wordpress, or will be a proxy to an existing feed reader service.
- Following private content. Private content has always been a challenge, mainly due to the fact that any time authentication is involved it complicates things a lot. With IndieAuth finally written up as a spec, we now have a solid building block to use to experiment in this area more.
- An Android Microsub app. While both Monocle and Together work pretty well on mobile browsers, there are still many advantages to having a native Android app! And it sounds like one is already in the works.
- Better UI for actually following people. I've kind of taken a shortcut on this front in order to move the rest forward. Right now, you still need to type someone's URL into a Microsub app in order to follow them. There are many challenges with streamlining this process further.
As always, I'm happy to chat about any and all of this! It's been a lot of fun already to build this all out and see it working! You can find me in the IndieWeb chat via IRC and Slack, if you send me a Webmention I'll see it in my reader, or find me at an upcoming IndieWeb event!
-
Sunlit 2.0 Released as a Micro.blog and WordPress Photo Blogging App – MacStories (www.macstories.net)
-
Just implemented mp-destination on my site! I can connect multiple Micropub or Twitter accounts to my site to be able to post to other accounts from Micropub clients when I'm logged in as me. This is the first step of turning Monocle into more of a Tweetdeck-like experience!
-
EdwardHinkle
https://github.com/EdwardHinkle
•
Feb 20
Yep, exactly. Or a specialized app like Teacup that sends ate and drank posts. Those should still go through regardless.
As you said, @manton, it's more of a suggestion but especially a suggestion for generalized Micropub application, as opposed to specialized.
-
manton
https://github.com/manton
•
Feb 20
@aaronpk I'm glad you mentioned Post Type Discovery, because to me that is the part to focus on. It seems too complicated to require spelling out every property, like category or photo. If an endpoint doesn't support accepting a category on a new post, no harm done.
What can we borrow from the Post Type Discovery spec that will help here? At the very least it seems like the Microformats class names should be consistent.
In my example (https://indieweb.org/Micropub-brainstorming#Query_for_supported_vocabulary) I included what I view as the common actions from an app like Indigenous: like-of, repost-of, and bookmark-of, but bookmarks aren't actually mentioned in Post Type Discovery. I wonder if they should be, or are they not different enough from a regular post to list separately?
Interesting, I didn't actually realize bookmark wasn't in Post Type Discovery. It looks like it was mentioned under "Other Types Under Consideration" before it was moved to the W3C repo. Now the W3C note links to the Kinds of Posts section on the IndieWeb wiki for that.
The way we were adding things to the list of types in the algorithm was roughly based on how well-established the markup was in the wild. I am kind of surprised bookmarks didn't make that cut, but oh well.
The one potential confusion here is that post types are not the same as
h-*types, e.g. there is noh-replybecause you use thein-reply-toproperty onh-entryto create a reply post. I think that just means we need to be explicit about what to call this. To build on your previous example, this could be a solution:{ "post-types": [ { "type": "note", "name": "Note" }, { "type": "article", "name": "Blog Post" }, { "type": "photo", "name": "Photo" }, { "type": "video", "name": "Video" }, { "type": "reply", "name": "Reply" }, { "type": "like", "name": "Like" }, { "type": "repost", "name": "Repost" }, { "type": "rsvp", "name": "RSVP" }, { "type": "bookmark", "name": "Bookmark" } ] }Clients should assume that if it's not in the list, then the server doesn't support it? Of course there needs to be some sensible behavior for servers that don't return this info at all.
Would it make sense to omit
notefrom this list since that's kind of a baseline? Or keep it in the list because it allows the client to customize the name of the button still? -
manton
https://github.com/manton
•
Feb 20
@EdwardHinkle Sounds great! I can implement it at any time on Micro.blog, although first I wonder if anyone has feedback on the JSON key names.
One thing to note is these are all properties of the
h-entryvocabulary, whereas other kinds of posts support totally different properties. Things likeh-eventorh-reviewwhere they are completely different things. Continuing down this path it would make sense to scope these properties to note that they are part of theh-entryvocabulary.The other question is how many of the supported properties would need to be defined? If not all of them, (e.g.
photo,category,published) why these ones in particular?Is it because these correspond with post types? If that's the case, then maybe this should be somehow tied to the Post Type Discovery spec which spells out what properties map to what kinds of posts.
In that case, it may make more sense to have the server advertise which of these post types it supports, and then clients would look at the spec to know which properties to send to create those posts.
-
New Micropub library + XRay
- curl https://example.com/micropub -X POST -d h=entry -d content="Hello World" \
- -d photo=http://example.com/foo.jpg -d category=foo -d published=2018-01-01T10:00:00-0800
- ---
- <?php
- require('vendor/autoload.php');
- $xray = new p3k\XRay();
- // Read the input as a string to handle both form-encoded and JSON requests
- $input = file_get_contents('php://input');
- // Parse the Micropub request
- $request = p3k\Micropub\Request::create($input);
- // Output as an Mf2 array
- $item = $request->toMf2();
- // Turn it into an Mf2 page
- $mf2 = ['items' => [$item]];
- // Process via XRay
- $parsed = $xray->process(false, $mf2);
- print_r($parsed['data']);
- ---
- Array
- (
- [type] => entry
- [published] => 2018-01-01T10:00:00-0800
- [category] => Array
- (
- [0] => foo
- )
- [photo] => Array
- (
- [0] => http://example.com/foo.jpg
- )
- [content] => Array
- (
- [text] => Hello World
- )
- )
-
Automatically POSSEing Likes and Reposts
My #newwwyear goal this year was to make my likes and reposts automatically POSSE to Twitter when I favorite a Tweet URL.
Previously, Quill handled doing that for me when I use its browser bookmarklet. If you connect your Twitter account with Quill, then if you favorite a tweet URL, it will go and favorite the tweet on Twitter for you. This works great if you only ever use Quill to favorite stuff. Now that I've been using Eddie's Indigenous iOS app, I had lost the ability to actually favorite stuff on Twitter if I posted it to my site.
My goal was to make my Micropub endpoint understand how to POSSE likes to Twitter so that no matter which client I used to favorite things, I'll end up favoriting the twitter copy too.
Thanks to silo.pub taking care of dealing with the Twitter API, it didn't end up being that much work to actually make this happen. So now, when my Micropub endpoint receives a request to create a like-of where the URL is a tweet URL, it tells silo.pub to also syndicate that post to Twitter.
So now when I encounter a tweet on iOS, I can use the "share" button to favorite it on my site which then also favorites it on Twitter!
So I was pretty excited about that, but then quickly realized I needed to go an extra step with it! Stretch goal! What happens when I favorite someone's indie post? It's great that it favorites it directly, but if they've also syndicated their post to Twitter, I should also syndicate my like to Twitter! Thankfully this was also not a huge amount of work, thanks to a lot of infrastructure already in place in my site.
When I create a like, my site already expands the contents of the liked post and stores it internally, as well as rendering the content I favorited. I like the way this ends up looking since it makes my favorites page look more like the equivalent page on Twitter.
Since my site is already pulling in the content of the post I like, that already includes any syndication URLs reported by that post. That means I already know the syndicated tweet URL that I need to favorite on Twitter!
Adding support for this was a matter of finding the Twitter syndication URL of the post I've favorited, and then telling silo.pub to favorite that on Twitter as well.
-
"Checked In By"
Swarm has had a feature for a long time where you can allow your friends to check you in. It's more than just tagging a friend in a checkin, since it actually creates a full-fledged checkin on the other person's account. It's an interesting feature of Swarm, and not something I've seen elsewhere.
Today I shipped an update to OwnYourSwarm which will include that information in the Micropub request it sends to your site.
Here's a checkin from last night that was created when anomalily checked me in.
Here's what it looks like on my site now.
To accomplish this, I added a new property in the Micropub JSON that OwnYourSwarm sends. The property is checked-in-by and the value is an h-card of the person who checked you in. There was some discussion about whether this should be shown with the person who checked you in as the author, but I argued against that since on Swarm the checkin appears in your own checkin history and is otherwise identical to a checkin you created yourself.
"checked-in-by": [ { "type": [ "h-card" ], "properties": { "url": [ "http://anomalily.net/", "https://foursquare.com/user/894076" ], "name": [ "anomalily" ], "photo": [ "https://igx.4sqi.net/img/user/300x300/894076-UEP315CWOZJNKG2J.jpg" ] }, "value": "http://anomalily.net/" } ]If you are using OwnYourSwarm's JSON format, you'll start receiving this additional property right away! If you've followed the Micropub spec and are already storing unrecognized properties in your storage, then you can later add support for rendering this property like I did!
Happy Swarming!
-
Example delete and undelete flow when storing posts as flat files:
• Create a post on disk stored at e.g. `/web/2017/10/15/1/post.txt` which has a public URL of `http://example.com/2017/10/15/1/`
• When the client deletes the post, the server moves the file to `/web/.trash/2017/10/15/1/post.txt`
• If the client sends an undelete request, the request will include the former public URL of the file `http://example.com/2017/10/15/1/`
• Similar to how the server knows where to find the file for live posts, it should know how to find the post in the trash folder if it's been deleted
• The server can move the file back to the public folder `/web/2017/10/15/1/post.txt` to complete the undelete
