### Generar embeddings para la consulta de búsqueda
[Sentence Transformers](https://www.sbert.net/) ofrece modelos de embedding locales y fáciles de usar para capturar el significado semántico de oraciones y párrafos.
El conjunto de datos de HackerNews contiene embeddings vectoriales generados con el modelo
[all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2).
A continuación se incluye un script de Python de ejemplo para mostrar cómo generar
vectores de embedding de forma programática usando el paquete de Python `sentence_transformers`. El vector de embedding de búsqueda se pasa luego como argumento a la función [`cosineDistance()`](/es/reference/functions/regular-functions/distance-functions#cosineDistance) en la consulta \`SELECT\`.
```python theme={null}
from sentence_transformers import SentenceTransformer
import sys
import clickhouse_connect
print("Initializing...")
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
chclient = clickhouse_connect.get_client() # ClickHouse credentials here
while True:
# Obtener la consulta de búsqueda del usuario
print("Enter a search query :")
input_query = sys.stdin.readline();
texts = [input_query]
# Ejecutar el modelo y obtener el vector de búsqueda
print("Generating the embedding for ", input_query);
embeddings = model.encode(texts)
print("Querying ClickHouse...")
params = {'v1':list(embeddings[0]), 'v2':20}
result = chclient.query("SELECT id, title, text FROM hackernews ORDER BY cosineDistance(vector, %(v1)s) LIMIT %(v2)s", parameters=params)
print("Results :")
for row in result.result_rows:
print(row[0], row[2][:100])
print("---------")
```
A continuación se muestra un ejemplo de la ejecución del script de Python anterior junto con los resultados de búsqueda por similitud
(solo se imprimen 100 caracteres de cada uno de los 20 posts principales):
```text theme={null}
Inicializando...
Ingrese una consulta de búsqueda :
Are OLAP cubes useful
Generando el embedding para "Are OLAP cubes useful"
Consultando ClickHouse...
Resultados :
27742647 smartmic:
slt2021: OLAP Cube is not dead, as long as you use some form of:1. GROUP BY multiple fi
---------
27744260 georgewfraser:A data mart is a logical organization of data to help humans understand the schema. Wh
---------
27761434 mwexler:"We model data according to rigorous frameworks like Kimball or Inmon because we must r
---------
28401230 chotmat:
erosenbe0: OLAP database is just a copy, replica, or archive of data with a schema designe
---------
22198879 Merick:+1 for Apache Kylin, it's a great project and awesome open source community. If anyone i
---------
27741776 crazydoggers:I always felt the value of an OLAP cube was uncovering questions you may not know to as
---------
22189480 shadowsun7:
_Codemonkeyism: After maintaining an OLAP cube system for some years, I'm not that
---------
27742029 smartmic:
gengstrand: My first exposure to OLAP was on a team developing a front end to Essbase that
---------
22364133 irfansharif:
simo7: I'm wondering how this technology could work for OLAP cubes.
An OLAP cube
---------
23292746 scoresmoke:When I was developing my pet project for Web analytics ( int:
encoding = tiktoken.encoding_for_model(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
chclient = clickhouse_connect.get_client(compress=False) # Credenciales de ClickHouse aquí
while True:
# Obtener la consulta de búsqueda del usuario
print("Enter a search topic :")
input_query = sys.stdin.readline();
texts = [input_query]
# Ejecutar el modelo y obtener el vector de búsqueda o de referencia
print("Generating the embedding for ----> ", input_query);
embeddings = model.encode(texts)
print("Querying ClickHouse...")
params = {'v1':list(embeddings[0]), 'v2':100}
result = chclient.query("SELECT id,title,text FROM hackernews ORDER BY cosineDistance(vector, %(v1)s) LIMIT %(v2)s", parameters=params)
# Concatenar todos los resultados de búsqueda
doc_results = ""
for row in result.result_rows:
doc_results = doc_results + "\n" + row[2]
print("Initializing chatgpt-3.5-turbo model")
model_name = "gpt-3.5-turbo"
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
model_name=model_name
)
texts = text_splitter.split_text(doc_results)
docs = [Document(page_content=t) for t in texts]
llm = ChatOpenAI(temperature=0, model_name=model_name)
prompt_template = """
Write a concise summary of the following in not more than 10 sentences:
{text}
CONSCISE SUMMARY :
"""
prompt = PromptTemplate(template=prompt_template, input_variables=["text"])
num_tokens = num_tokens_from_string(doc_results, model_name)
gpt_35_turbo_max_tokens = 4096
verbose = False
print("Summarizing search results retrieved from ClickHouse...")
if num_tokens <= gpt_35_turbo_max_tokens:
chain = load_summarize_chain(llm, chain_type="stuff", prompt=prompt, verbose=verbose)
else:
chain = load_summarize_chain(llm, chain_type="map_reduce", map_prompt=prompt, combine_prompt=prompt, verbose=verbose)
summary = chain.run(docs)
print(f"Summary from chatgpt-3.5: {summary}")
```