The Future of LLM: Exploring the Power of AutoGen

Introduction

In this blog we are going to talk about AutoGen. AutoGen is a toolkit that lets developers build LLM applications using multiple agents that can chat with each other to solve problems. AutoGen agents are customizable, can converse, and let people join in seamlessly. They can work in different ways, using different combinations of LLMs, human input, and tools. Some of sample example provided by the Developers of the AutoGen are shown below

AutoGen gives each agent its own instance of the LLM, so they can work independently. This helps us separate out the conversations that would normally take place in the same chat in ChatGPT or Bard. This way, we can assign each agent a separate role by providing specific system messages, as we have explored in this blog. This allows us to get better results from each agent, thereby improving the overall outcome of the generated content.

This also opens up new possibilities that were not previously available in other libraries, such as LangChain. In LangChain, we can assign a role to the LLM and ask it progressive questions, to which it replies. Now, consider a scenario where we can allow the LLMs to talk to each other so that they can progressively build on the conversation and provide a final output that could be even better.

For example, we can take the case of Multi-Agent Conversation that we described earlier. We can assign the role of developer to one agent, the role of code reviewer to another agent, and the role of user to provide any feedback to the agent's. Then, we can ask the developer agent to write a particular code. Instead of manually checking the code to see if it is correct, the code reviewer agent can check it and return suggestions to the developer agent to make any necessary changes. This will significantly improve the output quality of the code.

Example

To showcase the power of AutoGen we are going to develop a simple scenario where the teacher is assigning project to multiple students as a group project and we assign individual students some characteristic's by which they will work together to complete this project and in between and towards the end we have also a agent to assist the teacher in checking the code to see if it correct for the given project requirements.

To start with the project we first create a virtual environment to make sure we are not encountering any library issues. To create a virtual environment:

python -m venv pyautogen

Once the virtual environment is created activate the environment. Then install the following libraries

AutoGen:

pip install pyautogen

Jupyter Notebook

pip install notebook

Chroma DB

pip install chromadb

PyPDF

pip install pypdf>=3.1.0

Tik Token

pip install tiktoken

Once you have installed the above libraries we are ready to go, create a Jupyter notebook and import the following libraries:

import autogen
from autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent
from autogen import AssistantAgent
import chromadb

Then for us to use a AutoGen we have to define a variable called as config_list which we will be passing it to the agents where we will be specifying the LLM models we are planning to use in this application and other attributes. For using LLM models AutoGen currently support OpenAI models or Azure LLM models which are extension of OpenAI model available on Azure. There is currently no out of the box support for other LLM models such as PaLM by Google or open source models. To keep it cost effective we are using GPT 3.5 model specifically "gpt-3.5-turbo-16k-0613" model which is cheaper compared to the GPT-4 but the quality of the generation will also be affected. If you want better outputs I would recommend using GTP-4 model while taking cost into account.

For config_list it is recommended by the AutoGen developers to either set it though environment variable or load in through a JSON or YAML file. you can find more information about it in the official GitHub Repo. In our case since the config list was define as shown below

config_list = [{"model": "gpt-3.5-turbo-16k-0613", "api_key": "<your OpenAI API key here>"},]

Then we proceed to define llm_config where define some parameters for LLM.

llm_config = {
    "request_timeout": 60,
    "seed": 42,
    "config_list": config_list,
    "temperature": 0,
}

Now we define each agent

First Teacher:

teacher = autogen.UserProxyAgent(
    name="Teacher",
    is_termination_msg=termination_msg,
    human_input_mode="TERMINATE",
    system_message="The Teacher who assigns project and guides student on if the project is right or wrong",
    code_execution_config=False,  # we don't want to execute code in this case.
)

Now we add a assistant to teacher to provide dataset to the students in case they need them so we define a new agent as follows

teacher_aid = RetrieveUserProxyAgent(
    name="Teacher_Assistant",
    is_termination_msg=termination_msg,
    system_message="Assistant who has extra content retrieval power for providing dataset.",
    human_input_mode="TERMINATE",
    max_consecutive_auto_reply=3,
    retrieve_config={
        "task": "code",
        "docs_path": "https://raw.githubusercontent.com/yishuen/yelp-restaurant-recommendation-system/master/finaldata.csv",
        "chunk_token_size": 1000,
        "model": config_list[0]["model"],
        "client": chromadb.PersistentClient(path="/tmp/chromadb"),
        "collection_name": "groupchat",
        "get_or_create": True,
    },
    code_execution_config=False,  # we don't want to execute code in this case.
)

Now we go with a agent to check the code if it is right or wrong and should be able to execute the code to see the results.

teacher_check = autogen.AssistantAgent(
    name="Teacher_Check_Answer",
    is_termination_msg=termination_msg,
    system_message="Should check if the answer provided by the student is correct for the question/project asked by the teacher",
    llm_config=llm_config,
code_execution_config={"last_n_messages": 3, "work_dir": "project"},
)

Now we define the students who are in a group.

student1 = AssistantAgent(
    name="Student_1",
    is_termination_msg=termination_msg,
    system_message="Has very good knowledge on AI/ML related coding in Python but has no knowledge in R. Reply `TERMINATE` in the end when everything is done.",
    llm_config=llm_config,
)

student2 = autogen.AssistantAgent(
    name="Student_2",
    is_termination_msg=termination_msg,
    system_message="Has very good knowledge in R related coding for AI/ML tasks but not good in python. Reply `TERMINATE` in the end when everything is done.",
    llm_config=llm_config,
)

Now we define the problem that the teacher gives to both the students as a group

PROBLEM = "Perform Data Analysis using python on given dataset having user_id, business_id, rating as features with rating as\
the target variable. After performing Data Analysis which write a python code to make new features using pandas operations\
and then build a ml model using python and sci-kit learn library Once coded you will have to write the similar code in R using appropriate libraries"

Then we define the workflows of the conversation in our case we designed workflow based on Retrieval Augmented Generation and define all the potential workflow conversation that needs to keep place as we have explained earlier. Once the workflow has been designed we call the function to run the conversation. Here is sample conversation I have provided for the above example.

Teacher (to chat_manager):

Perform Data Analysis using python on given dataset having user_id, business_id, rating as features with rating as the target variable. After performing Data Analysis which write a python code to make new features using pandas operations and then build a ml model using python and sci-kit learn library Once coded you will have to write the similar code in R using appropriate libraries

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Teacher_Assistant (to chat_manager):



--------------------------------------------------------------------------------
Student_1 (to chat_manager):

Sure, I can help you with that. Could you please provide me with the dataset you have?

--------------------------------------------------------------------------------
Student_1 (to chat_manager):

Also, could you please specify what kind of data analysis you would like to perform on the dataset?

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Teacher_Assistant (to chat_manager):



--------------------------------------------------------------------------------
Student_1 (to chat_manager):

To perform data analysis on the given dataset, we can follow these steps:

1. Import the necessary libraries:
```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
```

2. Load the dataset:
```python
df = pd.read_csv('dataset.csv')
```

3. Explore the dataset:
```python
# Display the first few rows of the dataset
print(df.head())

# Check the shape of the dataset
print(df.shape)

# Check the data types of the columns
print(df.dtypes)

# Check for missing values
print(df.isnull().sum())

# Check the summary statistics of the dataset
print(df.describe())
```

4. Perform data preprocessing:
```python
# Drop any rows with missing values
df = df.dropna()

# Convert categorical variables to numerical using one-hot encoding
df = pd.get_dummies(df)

# Split the dataset into features and target variable
X = df.drop('rating', axis=1)
y = df['rating']
```

5. Perform exploratory data analysis (EDA):
```python
# Visualize the distribution of the target variable
sns.histplot(df['rating'])
plt.show()

# Visualize the correlation between features and the target variable
corr_matrix = df.corr()
sns.heatmap(corr_matrix, annot=True)
plt.show()
```

6. Feature engineering:
```python
# Create new features using pandas operations
df['feature1'] = df['column1'] + df['column2']
df['feature2'] = df['column3'] * df['column4']
```

7. Build a machine learning model:
```python
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize the model
model = LinearRegression()

# Fit the model to the training data
model.fit(X_train, y_train)

# Make predictions on the testing data
y_pred = model.predict(X_test)

# Evaluate the model
from sklearn.metrics import mean_squared_error, r2_score

mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print("Mean Squared Error:", mse)
print("R-squared:", r2)
```

This is a basic outline of how you can perform data analysis and build a machine learning model using Python and the scikit-learn library. Let me know if you need any further assistance or if you have any specific requirements.

Once you are done with the Python code, I can help you with the equivalent code in R.

--------------------------------------------------------------------------------
Teacher_Check_Answer (to chat_manager):

The code provided by the student looks correct and follows the steps mentioned by the teacher. However, without the actual dataset, it is not possible to verify if the code will work as expected. It is important to ensure that the dataset is loaded correctly and the necessary preprocessing steps are applied before building the machine learning model. Additionally, the code for feature engineering and model evaluation should also be implemented correctly.

--------------------------------------------------------------------------------
Student_2 (to chat_manager):

I agree with the teacher's feedback. The code provided by the student seems correct and follows the general steps for data analysis and machine learning in Python. However, without the actual dataset, it is not possible to verify if the code will work as expected.

If you have the dataset, please provide it so that we can further assist you in implementing the code and verifying its correctness.

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Teacher_Assistant (to chat_manager):



--------------------------------------------------------------------------------
Teacher_Check_Answer (to chat_manager):

Thank you for your feedback. I apologize for not providing the dataset earlier. Here is the dataset:

```
user_id,business_id,rating
1,101,4.5
2,102,3.8
3,103,4.2
4,104,3.5
5,105,4.0
```

Please use this dataset to implement the code and verify its correctness. Let me know if you have any further questions or need any assistance.

--------------------------------------------------------------------------------
Student_1 (to chat_manager):

Thank you for providing the dataset. Based on the dataset you provided, here is an updated version of the code to perform data analysis and build a machine learning model using Python and scikit-learn:

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Load the dataset
df = pd.read_csv('dataset.csv')

# Explore the dataset
print(df.head())
print(df.shape)
print(df.dtypes)
print(df.isnull().sum())
print(df.describe())

# Perform data preprocessing
df = df.dropna()
X = df.drop('rating', axis=1)
y = df['rating']

# Perform exploratory data analysis (EDA)
sns.histplot(df['rating'])
plt.show()

corr_matrix = df.corr()
sns.heatmap(corr_matrix, annot=True)
plt.show()

# Feature engineering
df['feature1'] = df['user_id'] + df['business_id']

# Build a machine learning model
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print("Mean Squared Error:", mse)
print("R-squared:", r2)
```

Please note that this code assumes that the dataset file is named 'dataset.csv' and is located in the same directory as the Python script. Make sure to adjust the file path if necessary.

Let me know if you have any further questions or need any additional assistance.

--------------------------------------------------------------------------------
Teacher_Check_Answer (to chat_manager):

The updated code provided by the student looks correct and follows the steps mentioned by the teacher. It loads the dataset, performs data preprocessing, explores the dataset through EDA, performs feature engineering, and builds a linear regression model using scikit-learn. The code also evaluates the model using mean squared error and R-squared.

However, it is important to note that the feature engineering step in the code provided by the student is just an example. In a real-world scenario, feature engineering would involve more complex operations and transformations based on the specific dataset and problem at hand.

Overall, the code provided by the student is a good starting point for performing data analysis and building a machine learning model using Python and scikit-learn.

--------------------------------------------------------------------------------

As you can see the output of each agent is not up to mark and the since we had defined a limit to the conversation as 12 it stopped after this and student 2 was not able provide the R code as needed by the teacher. It limit was created to stop the conversation if the conversation did not happed as expected as the model output varies and some time can be inconsistent as the above example.

By this example It is very clear that such powerful LLM can be used in a multi-agent framework to complete complex task by allowing such LLM to have conversation with each other and correct any mistakes it make and provide an output.

Some of potential use cases

1. Automate Business Workflow such as Document/Report Generations based on provided information

2. Review Business Ideas by assigning roles accordingly

3. Automate Content Generation

4. Conduct Online Analysis and Make Reports

Also with the help of code generation and code running capabilities we can automate code many small program creation tasks to help the developer deal with other major parts.

If you want to build such a product using AutoGen or want to consult us to get better idea feel free to contact us contact@codersarts.com