Utilize open-source tools (primarily on GitHub) to build automation tools (e.g. data pipelines, batch evaluations, report generation)
Responsibilities include:
- Writing reproducible experiment scripts and reports
- Consolidating key metrics (e.g. accuracy, latency, cost, hallucination rate)
- Proposing optimization recommendations based on experimental results
- LLM testing and evaluation
- Comparison of RAG (Retrieval-Augmented Generation) approaches
- Data preprocessing and annotation
- Feature engineering
- Fine-tuning of small models