<aside> 🤖 This document describes a consulting project for a news startup that aimed to automate the process of summarizing news articles. The project involved creating a custom finetuned GPT3 model, a web app for generating summaries, and a lightweight rating system. Challenges included data gathering, evaluation, and prompt engineering vs finetuning. Key lessons learned include the black box nature of finetuning GPT3 models, the democratization of building AI apps, and the potential for product opportunities in evaluation front-ends, human evaluation systems, and data munging/finetuning pipelines. The author also reflects on the potential for non-engineering personas to add AI-powered features to their products.

</aside>

Problem Statement

• Consulting for a news startup whose goal is to present different views of a topic (e.g. instead of reading 1 CNN article about a topic, can you present 15 different views from different articles to get a more holistic an unbiased source)
• Key part of their product is presenting summaries of news articles (up to 15 per topic) - right now done manually by humans reading and summarizing articles in a very specific style guide
• Goal was to see what % of this could be automated (while keeping a human in the loop for end edits) in order to 10x the amount of content created

Things delivered

1. Custom finetuned model (GPT3) that took in the text of the article and returned a summary of the article in a very specific style guide; optionally supported taking in just the URL of an article and scraping the text as well
2. Working end-to-end web app that enables end users to use and generate the AI summaries for eventual use in their content management system
3. Lightweight rating system enabling end users to rate results from the AI for future use in fine tuning and monitoring results
4. Testing scripts that enable batch testing of results against existing models

Challenges Faced

• Data gathering: the pipeline needed to take in article text; this was data that previously did not exist (only the article URL was stored) so to get a proper dataset, we needed to get the article text; to do this, we wanted to avoid scraping as it is prone to be flaky, dealing with IP blocks, etc. and tried a news API ($250/month); turns out the hit rate (given URL, what % of times did the actual article text come back) was extremely low (sub 10%); ended up using a python library (newspaper3k) and hit rate was 80%+!

• Evaluation: there is a tricky art here; we first did an eval in spreadsheets and by hand; then as we started doing more prompt/model runs, things got extremely messy; we also ended up building out a basic front-end component to get ratings from end-users - turns out using the google sheets API is also a pain → have to create a service key through google console, etc. etc. for what seems like it should’ve been a very basic thing to seutp

  

• Prompt engineering vs finetuning: this was a battle; at first we experimented with how far we could get just prompt engineering and found that we hit challenges with a) the context window size given the length of articles b) the quality was topping out and it was difficult stuffing more “hacks” at the end of prompts to get it to do the right thing; we ended up experimenting with finetuning and then later ChatGPT - we found that the best results came from finetuning but struggled from a black box syndrome and it’s unclear how much more data will really improve things in the future