Substack Decupled My Subscriber Growth

How I ran an A/B test to decide whether I should move to Substack

In October 2022, I wrote a blog post comparing what DALLE 2 could generate to what you would get from a Fiverr artist for $10. It hit the #1 spot on HackerNews, netting me around 50,000 page views. However, I only got 2 subscribers from it. Granted, the post was controversial, and it didn’t focus on a specific niche of people that would be interested in subscribing, but I refused to believe my writing was so bad as to warrant a 0.004% conversion rate.

Unwilling to take responsibility, I blamed my website host Squarespace for being poorly optimized for blogging. However, I didn’t know how to find evidence for my excuse, so I put my indignation on the backburner. That is, until I wrote my blog post about hiring people to sit behind me to make me productive. (It resulted in me tripling my productivity.)

I knew I had a banger, so I scratched my head to figure out how I could make use of the incoming virality. I remembered my qualm with Squarespace, and I realized this was the perfect opportunity to see if I could eke out a higher conversion rate by using a different blogging platform.

I thought the post was a good fit for the HackerNews community, where the format of posts is just a title that links out to a website. For example, here is what my submission looked like (you can see my hunch was right, it did pretty well 😉):

To test the effectiveness of Squarespace, I decided to pit it against the blogging choice du jour, Substack. Initially I submitted a link to my Squarespace site; after it hit the front page on HackerNews, I turned on an A/B test¹ that would send half of the visitors to my Substack blog instead. (This delayed response is what caused the discrepancy in the site visits data below.)

Here is what the data looked like after I ended the experiment:

I was astonished: I expected Substack to perform better because it was more tailored towards blogging, but not 11.5 times better! An order of magnitude difference in conversion rates is enormous.

Possible Explanations

Substack has multiple features to optimize conversion:

• A popup that appears once you’re halfway through reading the post that asks you to subscribe. 

• Where it does ask you to subscribe, if you already have a Substack account your email is pre-filled, which significantly decreases friction. 

• Multiple other automatic calls to subscribe on the page.

• A strong brand; a Substack feels like something worth subscribing to, as opposed to a random site in the wild west of the internet.

Squarespace, on the other hand, doesn't have these features (at least by default); in fact I find it increases user friction because once the user enters their email, they have to confirm their subscription from a confirmation email that can get lost to spam.

Comment Quality

Aside from the way better conversion rate, another thing I noticed was that the Substack comments were of substantially better quality, and were far more positive. On Squarespace, I had to delete several comments that looked like this:

I would imagine this is because you have to have an account on Substack, but Squarespace lets you comment anonymously. Substack also has a better comments section in general, with the ability to like comments. For some reason, Squarespace removed this in version 7.1.

Moving to Substack

Initially I chose Squarespace for its flexibility: it lets me host blogs with latex, code blocks, and even my own Javascript running in the page. However, this experiment convinced me that the flexibility wasn’t worth it. If Substack converted 25% better I might have brushed it off, but there is no getting around 1050%. Consequently, I will be moving my blog to Substack, and when friends ask me what platform they should use to start their blog I will recommend Substack by default.

1. In reality I wrote a server that would randomly redirect people to either my Squarespace or my Substack, and the moderator of HackerNews changed the post link to my server. Note: he said this is the only time such an experiment is allowed, so don’t try this at home!