LLMGeneratedCodePowerConsumption

This repository investigates the energy consumption and performance of coding solutions generated by large language models (LLMs). It provides a systematic approach to measuring and analyzing the computational costs of solutions written in Python, C++, and Java, including both baseline solutions (human-written) and those generated by three LLMs (ChatGPT 4o, OpenAI o1-mini and GitHub Copilot).

---

Repository Structure

Folders

Sheets

• LeetCodeBenchmark.xlsx: Contains the benchmark problems used in the study.
• Tasks_and_Tags.xlsx: Lists all problems and their associated tags.
• Prompt_Structure.png: A visual representation of the prompt setup used to instruct the LLMs.
• Grouping_of_Tags.xlsx: Groups tags into broader categories of programming tasks.

src

• data/: Stores baseline and LLM-generated solutions.
  • Four subfolders:
    • baseline/: Baseline coding solutions written by humans and taken from LeetCode.
    • 4o/, o1-mini/, github/: Solutions generated by LLMs.
  • Each subfolder organizes solutions by problem, with separate files for Python, C++, and Java implementations.
• measurements_perf/: Measurements obtained using perf on Linux.
• measurements_powermetrics/: Measurements obtained using powermetrics on macOS.
• Other Files: Scripts for measuring execution time of solutions in data/ and outputting results to the measurements_perf/ and measurements_powermetrics/ folders.

Results

• Subfolders for each language (C++, Java, Python) containing:
  • Folders for perf and powermetrics results. Each folder includes:
    • Per-problem subfolders with execution time and energy consumption graphs comparing the baseline and LLM-generated solutions.
  • Files (resultsC.json, resultsJava.json, resultsPython.json) aggregating all results in JSON format.

Analysis

• Jupyter Notebooks:
  • Save_All_Results_*.ipynb: Processes raw measurement data and saves results to the Results folder.
  • Other notebooks for:
    • Correlation analysis.
    • Generating heatmaps (stored in Heatmap/).
    • Creating graphs categorized by tags or problems (stored in Graphs_tags/ and Graphs_problems/).

---

Usage

Measurement

1. Use the scripts in src to measure execution time and energy consumption:
   • For Linux: Measurements with perf are saved in src/measurements_perf.
   • For macOS: Measurements with powermetrics are saved in src/measurements_powermetrics.

Result Aggregation

Run the Save_All_Results_*.ipynb notebooks in the Analysis folder to save processed data to the Results folder.

Analysis

Use the analysis notebooks in Analysis to:

• Explore correlations between energy consumption and performance metrics.
• Visualize data with heatmaps and other graphs.

---

Outputs

Performance and Energy Graphs

• Per-problem graphs comparing execution time and energy consumption for baseline and LLM-generated solutions.

JSON Results

• Aggregated results for Python, C++, and Java in the Results folder.

---

License

This project is licensed under the MIT License.