Hindsight is 20-20

Let's pause on the hub for now. Instead, I want you to evaluate my prompts in this chat. In particular, I'm looking to analyze whether the prompts were effective and efficient. What are some tactics or techniques (with examples) that could have made this a more efficient and productive session?

In this project, we built a prototype of a tracker for updates for the Rainbow 6 Siege video game across 4 chats, involving prototyping, visual designing, debugging, and auditing. From each chat session, I gave Claude the following prompt: "Let's pause on the hub for now. Instead, I want you to evaluate my prompts in this chat. In particular, I'm looking to analyze whether the prompts were effective and efficient. What are some tactics or techniques (with examples) that could have made this a more efficient and productive session?"

The responses from each chat have been uploaded to this project as files (Feedback 1, Feedback 2, Feedback 3, and Feedback 4).

My goal is to improve my prompting for the future. I want to develop techniques that deliver accurate results, minimize duplicate calls, increase speed in responses, and optimize on token usage.

Our task now is going to be summary and synthesis. I want you to do the following:

1. Evaluate all the feedback critically. Use high-quality external sources to validate (minimize time online, and document the sources used). Check to make sure the feedback is accurate and makes sense, especially in the context of my goals listed above. (no direct output to me needed)
2. Create a summarized, condensed Prompting Tactics document that synthesizes the main takeaways and key tactics. This should be a digestible but actionable document I can use to learn. Cite sources here. Include short explanations of each tactic, any limitations or considerations, an example of a positive use of the tactic and an example that would benefit from using the tactic.
3. In chat (not in the Prompting Tactics document), list out any feedback that was inaccurate in step 1, what you corrected it to, and where you found evidence for that change.
4. Create a second Prompt Evaluation Scorecard document. This should be a tool I can use to check my prompts against the tactics and techniques documented in the Prompting Tactics document. This should also be something I can pass through an AI so it can evaluate my prompts as well. The scorecard should be unambiguous. Any AI using the scorecard should have the lowest likelihood of error or hallucination possible.

Before executing these steps, please confirm your understanding

Thank you. This output is looking good. Before testing the scorecard, I want to run a few validation steps. All of these should use the same sources referenced in the Prompting Tactics document. All validation should make use of the external sources, not internal memory.

1. Do the tactics make sense compared to the literature? Was there any hallucination of any kind, either in the tactics or the examples? If there is, correct it in the Prompting Tactics document, and make the corresponding update in the scorecard document.
2. Is the current tactics list missing any high-value techniques or best practices that are included in the literature, even if it wasn't in the initial feedback? If so, update the tactics document and the scorecard to include these ideas.
3. Validate that the scorecard document matches the tactics document. If not, make the necessary changes to the scorecard.
4. Validate that the scorecard document can stand alone and doesn't require the tactics document to be comprehensible (no unexplained references). If not, make the necessary changes to the scorecard.

Before executing this work, are there any other validation steps you would recommend including?

Great. Let's dry-run the prompt scorecard. I want you to evaluate my first 3 prompts from this chat (do not count the Q/A responses. The first prompt starts with "In this project, we built a prototype...", the second starts with "Thank you. This output is looking good..." and the third starts with "One thing I noticed - the new tactics...")

Here's the process I'd like you to follow for each prompt. Follow this sequentially and return the output in a table format:

1. Critique the prompt yourself without using the scorecard. Create a short (4-5 sentence) evaluation. Include what worked, what could be better, and an overall rating (Ship it, one cleanup pass, high risk of rework)
2. Then, evaluate the prompt using the scorecard we created.
3. Finally, share a summary verdict of where the scorecard was strong vs where it was weak. Be self-critical; your feedback may not have been the best and the scorecard may have been better.

Prioritize accuracy for this ask. Include a validation run on the results before sharing them.

Before executing this, is there anything you need to clarify to ensure a successful outcome?

Your plan on 1-3 works for me. I understand the bias risk here. Do you have any suggestions for how to minimize this risk? I was not aware of any so was ok accepting the outcome, but open to suggestions

Let's proceed with techniques 1-5 applied. You are correct that the primary objective here is to test the scorecard mechanics.

These are exactly the kind of issues I was hoping to catch. Yes, we should make updates, but before doing so let's make sure we're aligned on what those updates need to be:

- T4/T6 interaction - this seems like the biggest weakness, as you pointed out. However, your proposed fix creates a bit of a recursive loop (T4 evaluation --> T6 evaluation --> recheck T4). I think a clean scorecard would eliminate that. To correct this, I suggest re-ordering the tactics and the scorecard such that current T6 becomes the new T4. I think that also improves the overall logical flow. What do you think? Let's discuss before executing.
- T5 - could you explain a little more what the distinction you're making here is? Maybe showcasing this using prompt 1 as an example would be helpful.
- I'm a little concerned about the statement that T1 and T6 lean on judgement. How can we make this more concrete? Open to suggestions here
    - For T1 - Perhaps removing the "concrete" language is good enough. That would mean the evaluation needs to check that "acceptance criteria are testable", which feels less judgement driven. Thoughts? Be critical here.
    - For T6 - similarly, what if we change the "partial" categorization to read "Format is stated for some outputs but not others; OR format is implied ("a summary")"

Aligned with the updates. One confirmation:

1. You correctly noted that the reorder direction impacts both files. When doing this work, you will a) not adjust any of the content, only the positions, and b) you will update this holistically through both documents (i.e. all sections and all references will be updated)? If this is correct, please execute

All other steps make sense to me. Please include a validation step to check your output after the updates are made.