compile with -O3 to increase tok/s from 18 to 98! wow, i have to train a bigger model now

README.md

@@ -5,7 +5,7 @@ Have you ever wanted to inference a baby [Llama 2](https://ai.meta.com/llama/) m
 
 <img src="assets/llama_cute.jpg" width="300" height="300">
 
-With this code you can train the Llama 2 LLM architecture from scratch in PyTorch, then save the weights to a raw binary file, then load that into one ~simple 500-line C file ([run.c](run.c)) that inferences the model, simply in fp32 for now. On my cloud Linux devbox a dim 288 6-layer 6-head model (~15M params) inferences at ~18 tok/s in fp32, and about the same on my M1 MacBook Air. I was somewhat pleasantly surprised that one can run reasonably sized models (few ten million params) at interactive rates with an approach this simple.
+With this code you can train the Llama 2 LLM architecture from scratch in PyTorch, then save the weights to a raw binary file, then load that into one ~simple 500-line C file ([run.c](run.c)) that inferences the model, simply in fp32 for now. On my cloud Linux devbox a dim 288 6-layer 6-head model (~15M params) inferences at ~100 tok/s in fp32, and about the same on my M1 MacBook Air. I was somewhat pleasantly surprised that one can run reasonably sized models (few ten million params) at highly interactive rates with an approach this simple.
 
 Please note that this is just a weekend project: I took nanoGPT, tuned it to implement the Llama-2 architecture instead of GPT-2, and the meat of it was writing the C inference engine in [run.c](run.c). As such, this is not really meant to be a production-grade library right now.
 
@@ -22,7 +22,7 @@ wget https://karpathy.ai/llama2c/model.bin -P out
 (if that doesn't work try [google drive](https://drive.google.com/file/d/1aTimLdx3JktDXxcHySNrZJOOk8Vb1qBR/view?usp=share_link)). Compile and run the C code:
 
 ```bash
-gcc -o run run.c -lm
+gcc -O3 -o run run.c -lm
 ./run out/model.bin
 ```
 
@@ -33,11 +33,11 @@ pip install sentencepiece
 python run_wrap.py
 ```
 
-You'll see text stream. On my M1 MacBook Air this runs at ~18 tokens/s, not bad for super naive fp32 single-threaded C code. Sample output:
+You'll see text stream. On my M1 MacBook Air this runs at ~100 tokens/s, not bad for super naive fp32 single-threaded C code. Sample output:
 
 *Once upon a time, there was a boy named Timmy. Timmy loved to play sports with his friends. He was very good at throwing and catching balls. One day, Timmy's mom gave him a new shirt to wear to a party. Timmy thought it was impressive and asked his mom to explain what a shirt could be for. "A shirt is like a special suit for a basketball game," his mom said. Timmy was happy to hear that and put on his new shirt. He felt like a soldier going to the army and shouting. From that day on, Timmy wore his new shirt every time he played sports with his friends at the party. Once upon a time, there was a little girl named Lily. She loved to play outside with her friends. One day, Lily and her friend Emma were playing with a ball. Emma threw the ball too hard and it hit Lily's face. Lily felt embarrassed and didn't want to play anymore.
 Emma asked Lily what was wrong, and Lily told her about her memory. Emma told Lily that she was embarrassed because she had thrown the ball too hard. Lily felt bad
-achieved tok/s: 18.746993347843922*
+achieved tok/s: 98.746993347843922*
 
 ## howto
 
@@ -65,7 +65,7 @@ wget https://karpathy.ai/llama2c/model.bin -P out
 Once we have the model.bin file, we can inference in C. Compile the C code first:
 
 ```bash
-gcc -o run run.c -lm
+gcc -O3 -o run run.c -lm
 ```
 
 You can now run it simply as