### 검색 쿼리용 임베딩 생성
[Sentence Transformers](https://www.sbert.net/)는 문장과 단락의 의미론적 의미를 파악하기 위한 임베딩(embedding) 모델을 로컬 환경에서 간편하게 사용할 수 있도록 제공합니다.
이 HackerNews 데이터셋에는 [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) 모델로 생성된 벡터 임베딩(vector embedding)이 포함되어 있습니다.
아래에 예시 Python 스크립트가 제공됩니다. 이 스크립트는 `sentence_transformers` Python 패키지를 사용하여 프로그래밍 방식으로 임베딩 벡터를 생성하는 방법을 보여줍니다. 생성된 검색 임베딩 벡터는 `SELECT` 쿼리에서 [`cosineDistance()`](/ko/reference/functions/regular-functions/distance-functions#cosineDistance) 함수의 인수로 전달됩니다.
```python theme={null}
from sentence_transformers import SentenceTransformer
import sys
import clickhouse_connect
print("Initializing...")
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
chclient = clickhouse_connect.get_client() # ClickHouse credentials here
while True:
# 사용자로부터 검색 쿼리 입력 받기
print("Enter a search query :")
input_query = sys.stdin.readline();
texts = [input_query]
# 모델을 실행하여 검색 벡터 획득
print("Generating the embedding for ", input_query);
embeddings = model.encode(texts)
print("Querying ClickHouse...")
params = {'v1':list(embeddings[0]), 'v2':20}
result = chclient.query("SELECT id, title, text FROM hackernews ORDER BY cosineDistance(vector, %(v1)s) LIMIT %(v2)s", parameters=params)
print("Results :")
for row in result.result_rows:
print(row[0], row[2][:100])
print("---------")
```
위 Python 스크립트를 실행한 예시와 유사도 검색 결과는 다음과 같습니다
(상위 20개 posts 각각에서 100자만 출력됩니다):
```text theme={null}
초기화 중...
검색어를 입력하세요 :
Are OLAP cubes useful
"Are OLAP cubes useful"에 대한 임베딩 생성 중
ClickHouse 쿼리 중...
결과 :
27742647 smartmic:
slt2021: OLAP Cube is not dead, as long as you use some form of:1. GROUP BY multiple fi
---------
27744260 georgewfraser:A data mart is a logical organization of data to help humans understand the schema. Wh
---------
27761434 mwexler:"We model data according to rigorous frameworks like Kimball or Inmon because we must r
---------
28401230 chotmat:
erosenbe0: OLAP database is just a copy, replica, or archive of data with a schema designe
---------
22198879 Merick:+1 for Apache Kylin, it's a great project and awesome open source community. If anyone i
---------
27741776 crazydoggers:I always felt the value of an OLAP cube was uncovering questions you may not know to as
---------
22189480 shadowsun7:
_Codemonkeyism: After maintaining an OLAP cube system for some years, I'm not that
---------
27742029 smartmic:
gengstrand: My first exposure to OLAP was on a team developing a front end to Essbase that
---------
22364133 irfansharif:
simo7: I'm wondering how this technology could work for OLAP cubes.
An OLAP cube
---------
23292746 scoresmoke:When I was developing my pet project for Web analytics ( int:
encoding = tiktoken.encoding_for_model(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
chclient = clickhouse_connect.get_client(compress=False) # ClickHouse 자격 증명을 여기에 입력하세요
while True:
# 사용자로부터 검색 쿼리를 입력받습니다
print("Enter a search topic :")
input_query = sys.stdin.readline();
texts = [input_query]
# 모델을 실행하여 검색 또는 참조 벡터를 생성합니다
print("Generating the embedding for ----> ", input_query);
embeddings = model.encode(texts)
print("Querying ClickHouse...")
params = {'v1':list(embeddings[0]), 'v2':100}
result = chclient.query("SELECT id,title,text FROM hackernews ORDER BY cosineDistance(vector, %(v1)s) LIMIT %(v2)s", parameters=params)
# 모든 검색 결과를 하나로 합칩니다
doc_results = ""
for row in result.result_rows:
doc_results = doc_results + "\n" + row[2]
print("Initializing chatgpt-3.5-turbo model")
model_name = "gpt-3.5-turbo"
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
model_name=model_name
)
texts = text_splitter.split_text(doc_results)
docs = [Document(page_content=t) for t in texts]
llm = ChatOpenAI(temperature=0, model_name=model_name)
prompt_template = """
Write a concise summary of the following in not more than 10 sentences:
{text}
CONSCISE SUMMARY :
"""
prompt = PromptTemplate(template=prompt_template, input_variables=["text"])
num_tokens = num_tokens_from_string(doc_results, model_name)
gpt_35_turbo_max_tokens = 4096
verbose = False
print("Summarizing search results retrieved from ClickHouse...")
if num_tokens <= gpt_35_turbo_max_tokens:
chain = load_summarize_chain(llm, chain_type="stuff", prompt=prompt, verbose=verbose)
else:
chain = load_summarize_chain(llm, chain_type="map_reduce", map_prompt=prompt, combine_prompt=prompt, verbose=verbose)
summary = chain.run(docs)
print(f"Summary from chatgpt-3.5: {summary}")
```