Some LLM apps generate Python code and the execute it, which can lead to RCE(remote code execution) vulnerabilities*1. In this blog, we will explore why we still needs to run Python. And one of this answer is calculation.

*1 You can try my intentionally vulnerable LLM app, find the details for the very end of this article.

ChatGPT

If you ask ChatGPT to perform calculation, you will see that it will craft a Python code, then run it and give you the correct result.

Experiment of LLM Calculation

I conducted an experiment to see just how poor they are at performing calculations.

Below are the results of using OpenAI's GPT-3.5 to GPT-4 to perform multiplication. In the graphs, "x" and "y" represent the numbers being multiplied. Correct results are plotted as blue dots, while incorrect results are shown as red dots.

gpt-3.5-turbo-1106 (Accuracy Rate: 70.1%)

gpt-4o-mini (Accuracy Rate: 74.8%)

gpt-4 (Accuracy Rate: 77.5%)

gpt-4-turbo (Accuracy Rate: 88.2%)

gpt-4o (Accuracy Rate: 88.5%)

Summary

• All models are accurate when calculating two-digit by two-digit multiplications

• But as the number of digits increases, they start making mistakes

• Since an LLM is a 'Language Model,' these results are not surprising

Conclusion

If your LLM apps needs to perform calculation; like calculating price, converting something, then you will need to prepare a Python execution environment. Here is some tips to secure your app.

• Limit the capability of the execution environment

  • No Internet

  • Execution Time Limit

• Do not store any sensitive information, SSH private keys, API tokens, password.

You can find my vulnerable LLM app which can execute the Python and allow you to access .

https://www.shinohack.me/shinollmapp/hannah/

Normal Usage

RCE through Prompt Injection