Prompt Learning
本人很懒，不多介绍
引言：
具体
角色扮演
给予更多信息
指南：
2
1.
使用分割号，避免提示词冲突
eg.{text}三引号，单引号，双引号、XML 标记、章节标题等
2.
3.
4.
1.
2.
3.
迭代：
摘要：
1.
推理：
1.
转换：
扩展：
聊天机器人：
https://www.bilibili.com/video/BV1VZ421q7t3
课程总结：
关于这个 prompt，我初步有两个想法：
一、用大模型 LLM+prompt 帮我生成指令。我前期的第五章工作用手写的代码计算，设置规则，生成指令。可以提供信息（物体类别、颜色、位置），设置 prompt 让大模型帮我尝试生成指令，最后做测试。
二、借鉴这篇论文的思路， 设计一套二维模型的流程，用现有的检测模型，例如轻量级的 Yolo 生成检测结果，以及空间计算信息，然后使用 LLM 设置 prompt 推理为目标中的哪一个
参考的论文 demo 结果：
1 | How many doors are there in this room? |
分析参考论文吗 prompt：
角色目标：
1 | SINGLE_TURN_MODE_SYSTEM_PROMPT = """You are a dialog agent that helps users to ground visual objects and answer questiosn in a 3D room scan using dialog. The user starts the conversation with some goal object in mind, your goals are: |
1 | SINGLE_TURN_MODE_SYSTEM_PROMPT = """ 您是一个对话代理，可以使用对话帮助用户在 3D 房间扫描中定位可视对象并回答问题。用户带着某个目标对象开始对话，你的目标是 |
1.
1 | COMMANDS: |
1 | 指令： |
2.
1 | 2. Finish Grounding: This command is termed "finish_grounding", with arguments: {"top_5_objects_scores": {"<object_id>": "<object_score>"}, "top_1_object_id": "<top_1_object_id>"}, where score is number between 0 and 1 that you need to decide based on all information to indicate how likely this object should be selected to match with the user query. |
1 | 2. 完成接地： 该命令称为 "finish_grounding"，参数为 {"top_5_objects_scores": {"<object_id>"： "< 对象_分数 >"}, "top_1_object_id"： "<top_1_object_id>"}，其中 score 是介于 0 和 1 之间的数字，您需要根据所有信息来决定该对象被选中与用户查询匹配的可能性有多大。 |
3.
1 | 3. Finish Grounding: This command is termed "finish_grounding", with arguments: {"top_5_objects_scores": {"<object_id>": "<object_score>"}, "top_1_object_id": "<top_1_object_id>"}, where score is number between 0 and 1 that you need to decide based on all information to indicate how likely this object should be selected to match with the user query. |
1 | 3. 完成接地： 该命令称为 "finish_grounding"，参数为 {"top_5_objects_scores": {"<object_id>"： "< 对象_分数 >"}, "top_1_object_id"： "<top_1_object_id>"}，其中 score 是介于 0 和 1 之间的数字，您需要根据所有信息来决定该对象被选中与用户查询匹配的可能性有多大。 |
prompt：
1 | # 模板一 <obj><location> |
1.
1 | To achieve your goals, we can create functions to generate structured instruction templates based on the provided bbox and ids using the given template styles. Here's a Python implementation: |
2.
1 | To achieve the goals, let's create a Python script to generate structured instruction templates based on the provided bbox and ids using the given template styles. Then, we'll format the output to be compatible with Python's json.loads() function. Here's the implementation: |
3.
1 | Based on the provided instructions and templates, here's how we can reason about the results: |
4.
1 | Based on the user input and the provided templates, here are the appropriate instructions for the target object (camera) based on the four templates: |
5.
Sure, let’s generate appropriate instructions based on all the objects and expand to create more colorful and diverse templates for each template style.
Object Descriptions:
- Mug (ID: 6)
- Laptop (ID: 5)
- Bowl (ID: 2)
- Camera (ID: 3)
- Can (ID: 4)
Template One:
- Mug (Object 1):
- “mug at bottom right”
- Laptop (Object 2):
- “laptop at top right”
- Bowl (Object 3):
- “bowl at top left”
- Camera (Object 4):
- “camera at bottom left”
- Can (Object 5):
- “can at center”
Template Two:
- Mug (Object 1):
- “bottom right mug”
- Laptop (Object 2):
- “top right laptop”
- Bowl (Object 3):
- “top left bowl”
- Camera (Object 4):
- “bottom left camera”
- Can (Object 5):
- “center can”
Template Three:
- Mug (Object 1):
- “mug to the right of can”
- Laptop (Object 2):
- “laptop to the left of mug”
- Bowl (Object 3):
- “bowl in front of laptop”
- Camera (Object 4):
- “camera behind bowl”
- Can (Object 5):
- “can at the top of laptop”
Template Four:
- Mug (Object 1):
- “1st mug from left”
- Laptop (Object 2):
- “2nd laptop from right”
- Bowl (Object 3):
- “3rd bowl from top”
- Camera (Object 4):
- “4th camera from bottom”
- Can (Object 5):
- “5th can from center”
These instructions describe the positional relationships of each object using all the four templates. They are diverse and colorful, providing various ways to describe the spatial relationships between the objects.
Prompt Learning