I'm excited to announce that Micropub is now a W3C Recommendation! It's been a long road, but we made it! This is the final stage in the W3C spec lifecycle, and means that the spec has gone through several stages of review, and all parts of the spec have been implemented by at least two people.
Micropub began in 2013 when I outlined a simple API to create blog posts and short notes for my website, and published it on the IndieWeb wiki. I implemented it on my server as well as in a few new client applications, and was quickly using it day-to-day. The main design goal with Micropub is that it should be easy to implement, and it should build on top of existing standards: OAuth 2.0 and the Microformats 2 vocabulary.
Micropub is also intended to be implemented incrementally. Rather than having to read, understand, and implement the whole spec, you can start by implementing the basics of just creating simple text posts. You can later expand your implementation to support more complex objects, as well as editing and deleting posts.
One of the benefits of supporting Micropub on your server is that it allows you to leverage other peoples' work in building an interface to create posts on your own website. By 2014, there were already six independent server implementations, and four client implementations other than my own.
Over the next years, more and more people built out Micropub support in their blogging systems, including plugins for WordPress, Known, Kirby and more. Many client implementations have popped up as well, for a variety of platforms including the Micro.blog iOS application, a Ruby web app, an XMPP chat bot, and more. I also wrote a Micropub proxy service OwnYourGram which imports your Instagram photos to your website, and another called OwnYourSwarm that does the same for Foursquare checkins.
In 2015, the Social Web Working Group decided to adopt the Micropub spec to fulfill the client-to-server API aspect of our charter.
After taking the initial spec from the IndieWeb wiki and writing it up in W3C format, I brought it to the working group as an Editor's Draft. After working on it within the group for a few months, we resolved to publish it as a Working Draft in January 2016.
Publishing a Working Draft is typically the first time the spec will be noticed by other groups within the W3C, as it's the first time a working group signals that a spec is intending to reach Recommendation status. Micropub went through 5 revisions of Working Drafts from January to July 2016, gathering feedback from implementers and incorporating it into the spec. The changes ranged from editorial clarifications, to changes that affect the implementations such as names of properties or response formats. You can see all of the changes in the Micropub working drafts here.
At the point the working group is satisfied that the spec is in good shape, the group can request that the spec be promoted to a Candidate Recommendation. To do so, the group must demonstrate that the spec has received "wide review" from people outside of the working group. We used GitHub Issues to get and track feedback, to make it easy to show the feedback received and how any feedback was incorporated into the spec. Publishing a Candidate Recommendation requires the approval of the W3C director.
The spec then stays in the Candidate Recommendation stage for a minimum of four weeks. This time is used to gather feedback from other W3C member companies, as well as gathering implementation reports from anyone interested in the spec. Implementation reports are meant to collect information on which parts of the spec are being implemented, in order to get a sense of how mature the spec is and how well it's written. While the implementation report may look like a checklist you have to complete, it's totally fine to submit it with only a few boxes checked. It's more of an evaluation of the spec than an evaluation of your implementation.
In the W3C Social Web Working Group, we set an intentionally high bar for our specs to graduate out of Candidate Recommendation. In order to be promoted from Candidate Recommendation to Proposed Recommendation, our group decided that each feature of the spec must be implemented by at least two independent implementations. This helps ensure that the spec really does encourage interop between implementations.
I created a test suite, micropub.rocks, that you can use to test your client and server implementations. The test suite helps you fill out the implementation report, and also is a great tool for debugging your application as you're building it out.
If you're building a server, micropub.rocks will pretend to be a client and will send you requests that hit all the edge cases of the spec. If you're building a client, you can use micropub.rocks as a server to test how well your client can create and edit posts.
At the point the working group has determined that all features of the spec have been implemented, and that the spec has been reviewed by people outside the working group, the group can decide to request that the spec be promoted to Proposed Recommendation. The spec can't have any substantive changes between Candidate Recommendation and Proposed Recommendation, but things like typo fixes are okay. The W3C director must approve the request to transition to Proposed Recommendation. The Proposed Recommendation step is one last chance to broadcast widely that the W3C believes the spec is ready to publish as a Recommendation, and gives the W3C Advisory Committee an opportunity to formally object to any aspect of the spec.
Micropub reached Proposed Recommendation status in April 2017.
The final step is to publish the spec as a W3C Recommendation. In order to be promoted from Proposed Recommendation to Recommendation, the W3C Advisory Committee and the Director must approve the request. Once a Recommendation, the spec is finished, and can only be changed by submitting Errata.
If you're building a blogging platform, you can allow your users choose from a wide variety of posting clients by implementing the Micropub spec.
If you're building a posting client and want it to work with many different server backends instead of hard-coding it to Twitter or other proprietary APIs, implement the Micropub spec and you'll quickly have people eager to start using the app!
I hope this post has been a useful overview of what I've been working on for the past several years, and gives you a bit of a sense of what it's like to work on specs at the W3C!
I think this is the first time in the 100days project that I've worked on a project that is not my own! Today I added support for JSON requests to Known's Micropub endpoint. I also added support for JSON checkins that OwnYourSwarm sends.
I tried writing as little code as I could, and changing as little as possible about how it worked, so essentially I am just extracting the properties it knows about from the JSON request to the variables the plugin expects to find. This does mean that a few Micropub JSON features are still not supported, such as sending HTML content (Known seems to strip HTML tags from all content sent to it), and Known doesn't provide a mechanism for storing arbitrary nested JSON objects. However, I was able to get it to pass tests 200, 201 and 203 from the micropub.rocks test suite, which is enough for basic support.
It also is able to create checkins from the payload that OwnYourSwarm sends! I also made it download the photo that is attached to a checkin, rather than hotlink the Foursquare image URL.
Since I don't have commit access to the Known repo, I sent a pull request to Known with these changes. I tested everything with a local Known installation. Hopefully benwerd or mapkyca can merge the PR soon!
Hopefully this improves people's experience using tools like OwnYourSwarm and OwnYourGram with Known!
Today I wrote up documentation on OwnYourSwarm. It actually took quite a bit longer than I expected to write everything up. The documentation walks through each component:
Rather than repeat any of the information here, I will just send you off to read the docs! Please let me know if you have any questions! I hope to see some more implementations of people receiving checkins via Micropub soon!
An interesting feature of the Swarm app is how it handles photos uploaded to checkins. If you check in and attach a photo, the checkin is actually created before the photo is uploaded. If you're on a spotty Internet connection, you'll see this because your checkin will exist and you'll get points for it, but there won't be a photo yet. The app will then continue to upload the photo separately, retrying if it fails. This is actually a really great app design on the part of Foursquare, but does lead to some tricks with the API.
Since OwnYourSwarm uses Foursquare's realtime API, it will receive a POST request almost immediately, often before the photo exists at the API. This means the initial Micropub request might be missing the photo.
Today I made OwnYourSwarm send a Micropub update request to update your post after the photo is uploaded. When you post a checkin, if there is no photo, then OwnYourSwarm queues a background job on a 15-second delay. It will then check after 15 seconds to see if a photo exists, and sends an update request with the photo URL if so. If still no photo is found after 15 seconds, it will wait another 30 seconds and try again. This continues for the following schedule: 15 seconds, 30 seconds, 1 minute, 2 minutes, 5 minutes, 10 minutes, 30 minutes, 1 hour. We'll see if this is too much polling, but the rate limits on Foursquare are relatively high. (500 requests per user per hour). This does mean that every checkin with intentionally no photo will be requested from Foursquare 8 times.
I had originally planned on using this same polling schedule to later pull back responses to your checkins (likes, and comments), but Ryan pointed out that I can probably use a simpler and more efficient polling schedule since the Foursquare API provides a method to return the last N checkins.
Beginning a slow project of updating the docs about the IndieAuth spec, today I started by updating a few pages on the wiki. Right now, most of the docs about IndieAuth (the spec), and how to use it, live across a variety of pages on the wiki, grouped together at https://indieweb.org/Category:IndieAuth.
My goal with the spec IndieAuth is to make it an extension of the OAuth 2.0 spec. It has always been based off of the OAuth 2.0 spec, but I've never written it up properly as an extension. It's always been just a series of how-to guides. Additionally, I had originally started by specifying all of the responses being form-encoded responses, whereas the OAuth 2.0 spec says all responses must be JSON format. I've been slowly converting my various apps that use IndieAuth to support both, based on the HTTP Accept header, although I haven't publicized this much yet.
Today's project was updating the existing IndieAuth documentation on the wiki to explicitly include the JSON versions as a valid response format.
Eventually I hope to write up IndieAuth as a formal extension to OAuth 2.0, in the same format that something like the Device Flow is an extension.
Today I added an option to OwnYourGram to opt in to receiving an h-card as the location property instead of a geo:// URI. On the dashboard, there is a setting to toggle between receiving a geo:// URI with file uploads, and an h-card location with a reference to the Instagram photo URL.
Since Micropub's form-encoded format doesn't support nested objects like h-card, we have to use the JSON format to send that. However in JSON format, we can't upload files, so we need to send just the image URL. Typically, a Micropub client would first upload the photo to the user's Media Endpoint and include the resulting URL in the JSON Micropub request. However since the photo already exists at a URL on Instagram's server, I just skipped the extra upload step and pass the endpoint the Instagram URL.
So if you want to start receiving the location as an h-card, you can opt in to this change on the OwnYourGram dashboard!
I finished all the client tests for creating posts, and launched them on micropub.rocks today!
Here's a video showing how I sign in to the test suite in Quill and make a post that passes the first test.
Here are the list of tests currently implemented. The numbers correspond with the equivalent test for servers.
Currently the test tool does not keep track of which tests you've passed, it just tells you whether you passed them. It's still a manual step to check off the boxes in the implementation report.
The good news is that Micropub.rocks passes its own tests!
On the last telcon of the W3C Social Web Working Group, we voted to request that Micropub move to PR, the next stage of publication within the W3C! This is the second to last step before Micropub becomes a W3C Recommendation!
Since it's sometimes confusing to remember all the stages that specs go through, here is a little diagram to help, taken from the new W3C Process Document.
Today I prepared the document for publication. This involved a last-minute pass through the document looking for obvious typos, adjusting the language around the exit criteria to be in the past tense (when this is published we will have exited the Candidate Recommendation stage!), and expanding the acknowledgements section. With any luck, we'll be publishing this next week!
Now that I have a basic pattern for supporting edits in Quill, I was able to quickly add support for editing reposts today. It works the same as yesterday's project of supporting edits of "likes", where the main "edit" page detects the "repost-of" property in the post and redirects to the repost interface. Not much else to say about today's project other than that!
Today in Things I Should Have Done Ages Ago... I finally started adding support to Quill for editing posts. I'm going to have to build this out one by one for each post interface Quill supports, so I started with the easiest one, favorites.
Eventually I will add an "edit" button to my bookmarklets, which is currently how I access most of Quill for writing posts, replies, bookmarks, etc.
(I used to have cute icons there, but they disappeared long ago during some syncing process in Chrome.)
The bookmarklet will have code like the below:
The "/edit?url=" route in Quill does a Micropub query for all source properties, and inspects which properties come back. Depending on which properties are present, Quill then redirects to the appropriate editing interface based on the type of post it detects. For example, if it sees a "like-of" property in the post, then it redirects to "/favorite" with the post's URL in the query string.
The "/favorite" page then does a Micropub query for the specific properties it will edit (in this case, only the "like-of" property), and fills the URL in the form. When you click "Save", it makes a Micropub edit request to replace the value of the "like-of" property.
I'll be making more (slow) progress on adding editing support to the rest of the interfaces!