๐ฌ LLM Esoteric Code BenchmarksMarch 20, 2026โ
Tests passing
Esoteric Task Generator
A Python library to generate tasks in esoteric programming languages for benchmarking LLMs. It includes customizable templates, random code generators, and validation utilities. This tool helps researchers create diverse and challenging benchmarks for evaluating language models.
What It Does
- Generate random programs in esoteric languages.
- Customizable templates for task generation.
- Validation utilities for syntactic correctness.
Installation
Install the required dependencies:
pip install pyparsing==3.1.1 pytest==7.4.2Usage
Generated Task:
++[>++<-]Source Code
import random
from pyparsing import Word, alphas, nums, Optional, ParseException
def generate_brainfuck_task(complexity):
"""
Generate a random Brainfuck program based on the given complexity.
Args:
complexity (int): Determines the length and complexity of the program.
Returns:
str: A randomly generated Brainfuck program.
"""
commands = ['>', '<', '+', '-', '.', ',', '[', ']']
program = ''.join(random.choices(commands, k=complexity))
return program
def validate_brainfuck_syntax(program):
"""
Validate the syntax of a Brainfuck program.
Args:
program (str): The Brainfuck program to validate.
Returns:
bool: True if valid, False otherwise.
"""
stack = []
for char in program:
if char == '[':
stack.append('[')
elif char == ']':
if not stack:
return False
stack.pop()
return len(stack) == 0
def generate_task(language='brainfuck', complexity=5):
"""
Generate a task in the specified esoteric language.
Args:
language (str): The target esoteric language (default: 'brainfuck').
complexity (int): The complexity level of the task (default: 5).
Returns:
str: The generated task as a string.
Raises:
ValueError: If the language is unsupported or complexity is invalid.
"""
if complexity <= 0:
raise ValueError("Complexity must be a positive integer.")
if language == 'brainfuck':
task = generate_brainfuck_task(complexity)
if not validate_brainfuck_syntax(task):
raise ValueError("Generated Brainfuck program has invalid syntax.")
return task
else:
raise ValueError(f"Unsupported language: {language}")
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Esoteric Task Generator")
parser.add_argument("--language", type=str, default="brainfuck", help="Target esoteric language (default: brainfuck)")
parser.add_argument("--complexity", type=int, default=5, help="Complexity level of the task (default: 5)")
args = parser.parse_args()
try:
task = generate_task(language=args.language, complexity=args.complexity)
print("Generated Task:")
print(task)
except ValueError as e:
print(f"Error: {e}")Community
Downloads
ยทยทยท
Rate this tool
No ratings yet โ be the first!
Details
- Tool Name
- esolang_task_generator
- Category
- LLM Esoteric Code Benchmarks
- Generated
- March 20, 2026
- Tests
- Passing โ
Quick Install
Clone just this tool:
git clone --depth 1 --filter=blob:none --sparse \ https://github.com/ptulin/autoaiforge.git cd autoaiforge git sparse-checkout set generated_tools/2026-03-20/esolang_task_generator cd generated_tools/2026-03-20/esolang_task_generator pip install -r requirements.txt 2>/dev/null || true python esolang_task_generator.py