WordPress AI Tutorial 2 – Generate Excerpt With LLM And Langchain

Posts Excerpt Improve SEO

A well-crafted excerpt at the beginning of a blog post serves as a crucial first impression for readers. These concise summaries give visitors an immediate preview of what to expect, helping them quickly decide if the content matches their interests. Beyond improving user engagement, excerpts also benefit SEO by providing search engines with a clear overview of the post’s topic. Additionally, excerpts appear in search results and archive pages before users click through to the full post, allowing them to quickly grasp the main points and decide whether to read more – this saves time and improves the overall user experience.

Leverage LLM To Generate Excerpt

Generating excerpts is time-consuming because you must digest and summarize the content concisely. Just copying and pasting the first few sentences of a post may not be a good summary and may hurt the user experience.

Fortunately, we live in the age of LLM. LLM is good at summarizing content.

Why Use Langchain Rather Than Official API?

While using the official large language models(LLM) API directly is a valid approach, Langchain offers several key advantages that make it a compelling choice for AI development:

1. Simplified Integration: Langchain provides a unified interface for working with multiple AI models and services, making it easier to switch between different providers.
2. Built-in Memory Management: The framework includes sophisticated memory handling capabilities, allowing for more complex conversational flows and context retention.
3. Composability: Langchain’s modular design lets developers easily combine different components like prompts, models, and output parsers into custom chains.
4. Structured Output Handling: The framework provides robust tools for parsing and formatting AI responses, reducing the need for custom post-processing code.
5. Cost Optimization: Langchain includes built-in features for token counting and usage tracking, helping developers optimize their API costs.

Implement With Python And Langchain

The code demonstrates using Langchain to generate excerpts for posts in different languages. You can find the full code on GitHub.

# Load Packages
import pymysql
import pandas as pd
import os
import re
from langchain_core.messages import HumanMessage
from langchain_anthropic import ChatAnthropic

def connect_mysql():
    '''
    Connect to MySQL
    '''
    conn = pymysql.connect(
        host='your_host',
        user='your_user',
        password='your_password',
        database='your_database',
        port=3306,
        cursorclass=pymysql.cursors.DictCursor
    )
    cursor = conn.cursor()
    return conn, cursor

def get_posts():
    '''
    Get posts with language code
    '''
    sql = '''
    SELECT *
    FROM wp_posts
    WHERE post_type = 'post'
    AND post_status = 'publish'
    ORDER BY post_date DESC
    '''
    conn, cursor = connect_mysql()
    cursor.execute(sql)
    list_cursor = list(cursor)
    result = pd.DataFrame(list_cursor)
    conn.close()
    return result

def update_excerpt_and_log(row, summary):
    '''
    Update the excerpt of the post
    '''
    sql = f"""
    UPDATE wp_posts 
    SET post_excerpt = '{summary}'
    WHERE ID = {int(row['ID'])}
    AND post_type = 'post';
    """
    conn, cursor = connect_mysql()
    cursor.execute(sql)
    conn.commit()
    conn.close()

def summarize_post(row):
    '''
    Summarize the post in English
    '''
    model = ChatAnthropic(model="claude-3-5-sonnet-20240620",
                          temperature=0.2, max_tokens=200,
                          anthropic_api_key=os.environ["ANTHROPIC_API_KEY"])
    prompt_template = """
    Please mimic the writing style of the article and create a concise summary in English, keeping it under 50 words. 
    Provide only the summary content without any explanations.
    Imagine you are the author and avoid starting with:
    - "This article..."
    - "In this post..."
    - "We discuss..."
    - "Let me introduce..."
    Here is the article content:
    {content}
    Assistant: """
    # Clean the content
    strip_content = row['post_content'].strip()
    strip_content = re.sub(r'<[^>]*>', '', strip_content)
    # Skip if the content is too short
    if len(strip_content) < 100:
        print(f'{row["post_title"]} is too short')
        return
    message = HumanMessage(content=prompt_template.format(
        content=strip_content))
    response = model.invoke([message])
    summary = response.content
    summary = summary.replace(',', '，')
    return summary

def main():
    '''
    Main function
    '''
    os.environ["ANTHROPIC_API_KEY"] = 'your_anthropic_api_key'
    wp_posts = get_posts()
    for i in range(len(wp_posts)):
        row = wp_posts.iloc[i]
        summary = summarize_post(row=row)
        if summary is None:
            continue
        print(f'{summary}')
        try:
            update_excerpt_and_log(row=row, summary=summary)
        except Exception as e:
            print(e)
        print(f'{i} / {len(wp_posts)}')

if __name__ == '__main__':
    main()