It's remarkable how quickly you can go from an idea to production with the help of AI tools.
Earlier this week, I had a thought: what could I learn about trending topics in Java if I analyzed accepted talks at Java conferences? Thanks to ChatGPT and Copilot, I was able to create the analysis and publish my Java Ecosystem Trends Report 2023 less than a day after getting the idea.
I had all the skills needed to write the scraper, categorize data, and analyze the result. What I didn't have was a week to do it all by hand. Working with AI tools, I could translate my idea into reality in a fraction of the time.
Here's how I used AI tools to ship my idea faster
Here's what I wanted to do: find out which topics were most frequently present in accepted conference sessions.
I had a rough idea of the steps I needed to take:
Scrape conference data from websites
Categorize the data into distinct topics
Analyze the data to find out the most common topics and any other interesting data
Write a report
Scrape data
I began by asking ChatGPT to write me a web scraper script in Node based on what I was trying to accomplish. I went back and forth between coding in my IDE to asking ChatGPT to improve certain aspects of the script until I had something that worked for all the conferences I wanted to analyze. The output was a JSON file with session titles and abstracts per conference.
Categorize data
Next, I asked ChatGPT to help me write a Node script that takes the JSON file as input and sends each title+abstract to the OpenAI chat completion API to get classified. Again, I had to go back and forth several times. First, to work on the prompt, then to batch the requests and show a progress indicator as the process took a long time. I used the new OpenAI function calling mechanism to ensure I got valid JSON as output. The output was a CSV file with 3 columns: conference, topic, and count.
Analyze data
I performed the analysis step in a Jupyter Notebook using Python, Pandas, and Seaborn. I worked with ChatGPT to quickly do the analysis and create charts. I liked having the Python code allowed me to verify the work (as opposed to using a more automated AI data analysis tool).
Write the report
With the analysis done, the last remaining part was drawing conclusions and writing up a report. I already had some takeaways, but I still asked ChatGPT to look at the list of most common topics to see if it could develop other insights. Most of the insights were aligned with what I was already thinking, and others weren't that valuable. But some of its insights helped spark more ideas for me.
Finally, I wrote the report and hit publish. The 2023 Java Ecosystem Trends Report was born. I purposefully published it early as an MVP instead of trying to make it perfect. It has allowed med to get great feedback and insights on how to make the categorization more robust for next time.
Key takeaways
Know what you're trying to accomplish before you start. If not, you'll end up on a wild goose chase.
Work in small steps. Give enough context and iterate until you get the result you need.
Verify the results. Just because something is a statistically probable answer to your question doesn't mean it's right. You should be able to stand behind what you create.
Ship early and gather feedback.