Cover Image

Contents

The Philosophy

Generative AI has completely transformed the way I work and learn. During the pre-ChatGPT days, I had a personal goal to learn one new programming language a year and that felt reasonable. I’d spend 1-2 months researching the language and learning the basics, another 2-3 months building a handful of throwaway projects, and the rest of the year learning more advanced topics. Throw in a full time job, family life, and a few hobbies and that was about all I could manage.

But with generative AI, this goal feels small. Pathetically small.

In just the last 3 months, I’ve learned JavaScript/TypeScript, Swift, and a little of nearly 20 other languages. I’ve learned React and SwiftUI, written scripts in C++, Go, and Rust, rebuilt my personal blog in Astro, and started creating animations with Manim.

Goals I would have set for the entire year now feel achievable next week. In fact, the goals I used to have don’t make sense anymore as generative AI has made me care a whole lot less about languages and frameworks. I don’t need to be an expert in any of these to be productive.

In December of 2022, Stack Overflow banned Generative AI claiming that the “average rate of getting correct answers from ChatGPT and other generative AI technologies is too low”. The ban applies to generating both questions and answers.

I’m not here to say that this is a bad decision. Stack Overflow has become popular for providing users with a genuine, curated experience and Generative AI is anything but genuine and curated. Without proper limitations, I’m sure mediocre AI content would overwhelm the site and bury quality answers.

Nonetheless, I view Stack Overflow’s decision to veer away from AI as an opportunity. The number of developers using AI on a daily basis will increase exponentially until every developer is using it. It will be ubiquitous. Generative AI, despite its decreasing set of flaws, is an always on, lightning fast brain in a jar and to prevent people from becoming familiar with it seems like a long-term mistake.

To that end, I am creating Code on the Rocks(COTR), a Stack Overflow alternative that embraces AI. Initially, the goal will simply be to do what SO is not doing and allow answers generated by AI. Users can submit answers generated by ChatGPT, Gemini Pro, Claude, Perplexity, or any of the other popular LLMs that have descended into reality. The website will callout that generated answers are not always accurate and provide ways for users to vote on the quality of answers.

You might be wondering what, if any, problem this solves. I think there are a few:

  1. AI conversations are closed source. While almost anyone can chat with the big models, the average user has zero visibility into the conversations that other people are having with AI. Instead of learning and building in isolation, developers can share how they arrived at their final solutions by sharing chats.
  2. Subscriptions are expensive. Every formidable model on the market has a pro version available for $20/month so many people limit themselves to just one. COTR could provide a place to view crowd sourced answers from all of the big LLMs for a discounted bundle price.
  3. Stack Overflow isn’t perfect. It takes 16 minutes on average to receive an answer and about a quarter of questions go unanswered. AI has the potential to address both of these, providing intsant answers to all questions (accuracy currently optional)

For the average developer, the third problem is the most visceral and therefore the one I will focus on.

The Software

I started building the Code on the Rocks platform on March 20th, 2024 (a week ago today).

alt text

The initial version is built with the following technologies:

  • TypeScript
  • React (TailwindCSS, DaisyUI, React Query, React Router)
  • Supabase (pgvector)
  • Firebase (Hosting)
  • Google Analytics
  • Amplitude Analytics
  • Gemini Pro (Text completion, embeddings)

I chose all of these because I’ve used them before and wanted to build the MVP as fast as possible. The fact that I didn’t have to spend any money to create the website was also a huge factor. TypeScript, React, DaisyUI, TailwindCSS, React Query, React Router, and Supabase are all open source. Supabase has a generous free tier that should support me for while. Same for Amplitude and Firebase. Gemini Pro is also free for the time being and the quality of its answers is high enough. My total bill so far is $0.

Well, sort of.

I also relied heavily on Claude Opus and GitHub Copilot, both of which I subscribe to ($20 and $10, respectively). Claude was amazing at scaffolding pages (ex. “I need a NewQuestion.tsx file and a QuestionDetails.tsx file to display the information”) and I did at least 25% of my chatting with it while at the gym.

The website resembles Stack Overflow in design and is responsive across all screen sizes:

alt text

alt text

alt text

I have plans to eventually add Android and iOS apps built with Flutter but for the forseeable future, the “app” will be web-only.

The Numbers

The website is rigged to trigger a whole bunch of Amplitude events and is connected to Google Analytics:

  • Days Since Launch: 7
  • Total Questions: 72
  • Total Answers: 32
  • Total Sign Ups: 7
  • Total Users: 101

My One Metric That Matters (OMTM) is the number of questions asked per day. My goal with tracking this metric is twofold:

  • Measure user engagement
  • Determine if developers are willing to ask AI more questions than humans

In May of 2017, the Stack Overflow team mentioned in a blog post that developers ask “over 8,000 questions” on a typical day. This seems like a good goal to shoot for. I’m currently averaging about 10.

The Business

As with any other startup, the hardest part is marketing and finding the initial set of users. For this first week, my only acquisition channels have been Twitter and Threads and the results have been about as good as you’d expect (not good). While I did get a few kind folks to test out the platform, I also received a handful of comments like this:

./

As Sam Altman said in the first lecture on How to Start a Startup:

The best ideas often look terrible at the beginning

I’m taking this feedback as a sign I’m on the right track.

My goal for the next week is to clarify the positioning of the company so the audience is more targeted. Rather than target all developers with questions, I will be progressively narrowing the audience:

  • People who are just starting to learn programming
    • New programmers who are learning TypeScript
      • New TypeScript programmers who do not want to pay for AI tools

I will also be adding several new models that are available to answer coding questions, including Claude Opus and the Perplexity API. Happy coding 🍹