Build RNN-Transformer hybrid models

Build RNN+Transformer hybrid with O(n) inference using RWKV architecture

Combining RNN efficiency with Transformer quality when linear-time inference matters.

• · model config
• · training data
• · inference input

• · trained RWKV model
• · inference output
• · memory usage metrics

• · rwkv
• · torch
• · transformers

• · GPU available
• · PyTorch 2.0+

• · training slower than Transformers
• · inference unstable if learning rate too high

• · O(n) complexity proven
• · quality comparable to Transformers on benchmarks