Automating classification by using LLM outputs as features
Problem
Users want to use LLM judgments as features in downstream models for a more efficient and controllable classification pipeline; they also seek better prompting strategies to improve performance and reduce reliance on direct LLM classification.
Pain points
- Desire to automate a two‑LLM pipeline (strong for prompt generation, weak for classification) to save compute
- Desire to use LLM decisions as features in downstream classic ML models
- Need for a comparison or improvement using megaprompts to enhance feature quality
Questions to verify
- Are you interested in implementing a two‑LLM prompting pipeline in a real project?
- Are you currently using LLM decisions as features? What are the main challenges you face?
- Which prompting strategy do you think would most improve the quality of LLM outputs used as features?
Evidence
quote · @Anonymous
I think that this can be automated by using two LLMs: a stronger/more expensive for generating prompts and a weaker for actual classification. Approximate algorithm:1. Give "strong" LLM the task formulation and some labeled examples. Ask it to generate a prompt for the "weak" LLM.2. Run "weak" LLM on the training set with generated prompt from 1, use replies as features for a smaller ML model (logreg, decision tree etc).3. Pick examples from the training set that your small model is most wrong about and ask "strong" LLM to generate one more prompt (like in 1), except this time you are using the misclassified examples instead of random.4. Run "weak" LLM on generated prompt from 3, add results as one more feature for your model.5. Repeat 2 - 4 until your token budget for this task is ex
quote · @Anonymous
In my work on LLM as a judge, I prefer to use LLM decisions as features in a downstream classic ML model for the final decision.
quote · @Anonymous
Why does he first ask to label "ironic" or "not", and then answer the feature questions? Wouldn't it be better to reverse the order?
quote · @Anonymous
Really needs a comparison to the "megaprompt" itself (i.e. "here is a tweet, rate it as ironic or not, considering the following properties; explain your reasoning then output your final answer at the end"). I bet that would get you very far towards the logistic classifier, and would generalize much better out of distribution.
Disclaimer: this card is distilled by AI from community discussions as “pain point evidence”. It is a starting point for need analysis and validation, not a “validated market need”. AI has analytical limits; please open the original discussion above to manually verify before key decisions.