搜索引擎技术解读
2025/12/3大约 8 分钟
淘票票项目Elasticsearch实现技术文档
1. 技术概述
在淘票票项目中，Elasticsearch作为搜索引擎核心组件，提供高效的数据检索能力。本项目通过自定义的elasticsearch-framework模块封装了Elasticsearch的操作，实现了索引管理、数据CRUD和复杂查询等功能。
2. 项目实现架构
2.1 模块结构
taoppiao-elasticsearch-framework/
├── src/main/java/com/taopiaopiao/
│ ├── conf/ # 配置相关类
│ │ ├── BusinessEsAutoConfig.java # Elasticsearch自动配置类
│ │ └── BusinessEsProperties.java # Elasticsearch配置属性类
│ ├── dto/ # 数据传输对象
│ │ ├── EsDataCreateDto.java # 创建数据DTO
│ │ ├── EsDataQueryDto.java # 查询条件DTO
│ │ ├── EsDocumentMappingDto.java # 文档映射DTO
│ │ ├── EsGeoPointDto.java # 地理位置查询DTO
│ │ └── EsGeoPointSortDto.java # 地理位置排序DTO
│ └── util/ # 工具类
│ └── BusinessEsHandle.java # Elasticsearch核心操作类3. 核心配置实现
3.1 配置属性类 (BusinessEsProperties)
package com.taopiaopiao.conf;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
/**
* @program: 高度还原淘票票网高并发实战项目
* @description: elasticsearch配置属性
* @author: GGBOND
**/
@Data
@ConfigurationProperties(prefix = BusinessEsProperties.PREFIX)
public class BusinessEsProperties {
public static final String PREFIX = "elasticsearch";
private String[] ip; // Elasticsearch集群IP地址数组
private String userName; // 用户名
private String passWord; // 密码
private Boolean esSwitch = true; // Elasticsearch开关
private Boolean esTypeSwitch = false; // 是否使用类型（兼容旧版本）
private Integer connectTimeOut = 40000; // 连接超时时间
private Integer socketTimeOut = 40000; // Socket超时时间
private Integer connectionRequestTimeOut = 40000; // 连接请求超时时间
private Integer maxConnectNum = 400; // 最大连接数
}3.2 自动配置类 (BusinessEsAutoConfig)
package com.taopiaopiao.conf;
import com.taopiaopiao.util.BusinessEsHandle;
import org.apache.http.Header;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.nio.reactor.IOReactorConfig;
import org.apache.http.message.BasicHeader;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import java.util.Arrays;
import java.util.Objects;
/**
* @program: 高度还原淘票票网高并发实战项目
* @description: elasticsearch配置
* @author: GGBOND
**/
@EnableConfigurationProperties(BusinessEsProperties.class)
@ConditionalOnProperty(value = "elasticsearch.ip")
public class BusinessEsAutoConfig {
private static final int ADDRESS_LENGTH = 2;
private static final String HTTP_SCHEME = "http";
@Bean
public RestClient businessEsRestClient(BusinessEsProperties businessEsProperties) {
String defaultValue = "default";
HttpHost[] hosts = Arrays.stream(businessEsProperties.getIp()).map(this::makeHttpHost).filter(Objects::nonNull)
.toArray(HttpHost[]::new);
RestClientBuilder builder = RestClient.builder(hosts);
String userName = businessEsProperties.getUserName();
String passWord = businessEsProperties.getPassWord();
// 配置用户名密码认证
if (StringUtil.isNotEmpty(userName) && !defaultValue.equals(userName) && StringUtil.isNotEmpty(passWord) && !defaultValue.equals(passWord)) {
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, passWord));
builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
.setDefaultIOReactorConfig(IOReactorConfig.custom()
// 设置线程数
.setIoThreadCount(businessEsProperties.getMaxConnectNum())
.build()));
}
// 设置默认请求头
Header[] defaultHeaders = { new BasicHeader(HttpHeaders.CONTENT_TYPE, "application/json"),
new BasicHeader("Role", "Read") };
builder.setDefaultHeaders(defaultHeaders);
// 设置请求配置
builder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
.setConnectTimeout(businessEsProperties.getConnectTimeOut())
.setSocketTimeout(businessEsProperties.getSocketTimeOut())
.setConnectionRequestTimeout(businessEsProperties.getConnectionRequestTimeOut()));
return builder.build();
}
// 创建BusinessEsHandle实例
@Bean
public BusinessEsHandle businessEsHandle(@Qualifier("businessEsRestClient")RestClient businessEsRestClient, BusinessEsProperties businessEsProperties){
return new BusinessEsHandle(businessEsRestClient,businessEsProperties.getEsSwitch(),businessEsProperties.getEsTypeSwitch());
}
/**
* 获取HttpHost对象
* @param s 地址
*/
private HttpHost makeHttpHost(String s) {
assert StringUtil.isNotEmpty(s);
String[] address = s.split(":");
if (address.length == ADDRESS_LENGTH) {
String ip = address[0];
int port = Integer.parseInt(address[1]);
return new HttpHost(ip, port, HTTP_SCHEME);
} else {
return null;
}
}
}4. 核心功能实现
4.1 Elasticsearch操作核心类 (BusinessEsHandle)
BusinessEsHandle是项目中操作Elasticsearch的核心类，封装了索引管理、数据CRUD、查询等功能。
4.1.1 索引管理功能
/**
* 创建索引
*
* @param indexName 索引名字
* @param indexType 索引类型
* @param list 参数集合
*/
public void createIndex(String indexName, String indexType, List<EsDocumentMappingDto> list) throws IOException {
if (!esSwitch) {
return;
}
if (CollectionUtil.isEmpty(list)) {
return;
}
IndexRequest indexRequest = new IndexRequest();
XContentBuilder builder = JsonXContent.contentBuilder().startObject().startObject("mappings");
if (esTypeSwitch) {
builder = builder.startObject(indexType);
}
builder = builder.startObject("properties");
for (EsDocumentMappingDto esDocumentMappingDto : list) {
String paramName = esDocumentMappingDto.getParamName();
String paramType = esDocumentMappingDto.getParamType();
if ("text".equals(paramType)) {
Map<String,Map<String,Object>> map1 = new HashMap<>(8);
Map<String,Object> map2 = new HashMap<>(8);
map2.put("type","keyword");
map2.put("ignore_above",256);
map1.put("keyword",map2);
builder = builder.startObject(paramName).field("type", "text").field("fields",map1).endObject();
}else {
builder = builder.startObject(paramName).field("type", paramType).endObject();
}
}
if (esTypeSwitch) {
builder.endObject();
}
builder = builder.endObject().endObject().startObject("settings").field("number_of_shards", 3)
.field("number_of_replicas", 1).endObject().endObject();
indexRequest.source(builder);
String source = indexRequest.source().utf8ToString();
log.info("create index execute dsl : {}",source);
HttpEntity entity = new NStringEntity(source, ContentType.APPLICATION_JSON);
Request request = new Request("PUT","/"+ indexName);
request.setEntity(entity);
request.addParameters(Collections.<String, String>emptyMap());
Response performRequest = restClient.performRequest(request);
}
/**
* 检查索引是否存在
*
* @param indexName 索引名字
* @param indexType 索引类型
* @return boolean
*/
public boolean checkIndex(String indexName, String indexType) {
if (!esSwitch) {
return false;
}
try {
String path = "";
if (esTypeSwitch) {
path = "/" + indexName + "/" + indexType + "/_mapping?include_type_name";
}else {
path = "/" + indexName + "/_mapping";
}
Request request = new Request("GET", path);
request.addParameters(Collections.<String, String>emptyMap());
Response response = restClient.performRequest(request);
String result = EntityUtils.toString(response.getEntity());
System.out.println(JSON.toJSONString(result));
return "OK".equals(response.getStatusLine().getReasonPhrase());
}catch (Exception e) {
if (e instanceof ResponseException && ((ResponseException)e).getResponse().getStatusLine().getStatusCode() == RestStatus.NOT_FOUND.getStatus()) {
log.warn("index not exist ! indexName:{}, indexType:{}", indexName,indexType);
}else {
log.error("checkIndex error",e);
}
return false;
}
}
/**
* 删除索引
*
* @param indexName 索引名字
* @return boolean
*/
public boolean deleteIndex(String indexName) {
if (!esSwitch) {
return false;
}
try {
Request request = new Request("DELETE", "/" + indexName);
request.addParameters(Collections.<String, String>emptyMap());
Response response = restClient.performRequest(request);
return "OK".equals(response.getStatusLine().getReasonPhrase());
}catch (Exception e) {
log.error("deleteIndex error",e);
}
return false;
}4.1.2 数据添加功能
/**
* 添加数据
*
* @param indexName 索引名字
* @param indexType 索引类型
* @param params 参数 key:字段名 value:具体值
* @param id 文档id 如果为空，则使用es默认id
* @return boolean
*/
public boolean add(String indexName, String indexType,Map<String,Object> params, String id) {
if (!esSwitch) {
return false;
}
if (CollectionUtil.isEmpty(params)) {
return false;
}
try {
String jsonString = JSON.toJSONString(params);
HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
String endpoint = "";
if (esTypeSwitch) {
endpoint = "/" + indexName + "/" + indexType;
}else {
endpoint = "/" + indexName + "/_doc";
}
if (StringUtil.isNotEmpty(id)) {
endpoint = endpoint + "/" + id;
}
log.info("add dsl : {}",jsonString);
Request request = new Request("POST",endpoint);
request.setEntity(entity);
request.addParameters(Collections.<String, String>emptyMap());
Response indexResponse = restClient.performRequest(request);
String reasonPhrase = indexResponse.getStatusLine().getReasonPhrase();
return "created".equalsIgnoreCase(reasonPhrase) || "ok".equalsIgnoreCase(reasonPhrase);
}catch (Exception e) {
log.error("add error",e);
}
return false;
}4.1.3 数据查询功能
项目支持多种查询方式，包括普通查询、分页查询、地理位置查询和排序等功能。
/**
* 查询(分页)
*
* @param indexName 索引名字
* @param indexType 索引类型
* @param esGeoPointDto 经纬度查询参数
* @param esDataQueryDtoList 参数
* @param sortParam 排序参数 不排序则为空
* @param geoPointDtoSortParam 经纬度排序参数
* @param sortOrder 升序还是降序，为空则降序
* @param pageNo 页码
* @param pageSize 页大小
* @param clazz 返回的类型
* @return PageInfo
* @throws IOException
*/
public <T> PageInfo<T> queryPage(String indexName, String indexType, EsGeoPointDto esGeoPointDto,
List<EsDataQueryDto> esDataQueryDtoList, String sortParam,
EsGeoPointSortDto geoPointDtoSortParam, SortOrder sortOrder, Integer pageNo,
Integer pageSize, Class<T> clazz) throws IOException {
List<T> list = new ArrayList<>();
PageInfo<T> pageInfo = new PageInfo<>(list);
pageInfo.setPageNum(pageNo);
pageInfo.setPageSize(pageSize);
if (!esSwitch) {
return pageInfo;
}
// 构建查询条件
SearchSourceBuilder sourceBuilder = getSearchSourceBuilder(esGeoPointDto,esDataQueryDtoList,sortParam,geoPointDtoSortParam,sortOrder);
// 设置分页参数
sourceBuilder.from((pageNo - 1) * pageSize);
sourceBuilder.size(pageSize);
// 执行查询
executeQuery(indexName,indexType,list,pageInfo,clazz,sourceBuilder,null);
return pageInfo;
}
// 构建搜索条件
private SearchSourceBuilder getSearchSourceBuilder(EsGeoPointDto esGeoPointDto, List<EsDataQueryDto> esDataQueryDtoList,
String sortParam, EsGeoPointSortDto geoPointDtoSortParam, SortOrder sortOrder){
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
if (Objects.isNull(sortOrder)) {
sortOrder = SortOrder.DESC;
}
// 设置排序
if (StringUtil.isNotEmpty(sortParam)) {
FieldSortBuilder sort = SortBuilders.fieldSort(sortParam);
sort.order(sortOrder);
sourceBuilder.sort(sort);
}
// 设置地理位置排序
if (Objects.nonNull(geoPointDtoSortParam)) {
GeoDistanceSortBuilder sort = SortBuilders.geoDistanceSort("geoPoint",
geoPointDtoSortParam.getLatitude().doubleValue(),
geoPointDtoSortParam.getLongitude().doubleValue());
sort.unit(DistanceUnit.METERS);
sort.order(sortOrder);
sourceBuilder.sort(sort);
}
// 设置地理位置查询
if (Objects.nonNull(esGeoPointDto)) {
QueryBuilder geoQuery = new GeoDistanceQueryBuilder(esGeoPointDto.getParamName())
.distance(Long.MAX_VALUE, DistanceUnit.KILOMETERS)
.point(esGeoPointDto.getLatitude().doubleValue(), esGeoPointDto.getLongitude().doubleValue())
.geoDistance(GeoDistance.PLANE);
sourceBuilder.query(geoQuery);
}
// 构建布尔查询
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
for (EsDataQueryDto esDataQueryDto : esDataQueryDtoList) {
String paramName = esDataQueryDto.getParamName();
Object paramValue = esDataQueryDto.getParamValue();
Date startTime = esDataQueryDto.getStartTime();
Date endTime = esDataQueryDto.getEndTime();
boolean analyse = esDataQueryDto.isAnalyse();
// 处理普通查询条件
if (Objects.nonNull(paramValue)) {
if (paramValue instanceof Collection) {
// 集合类型处理
if (analyse) {
BoolQueryBuilder builds = QueryBuilders.boolQuery();
Collection<?> collection = (Collection<?>)paramValue;
for (Object value : collection) {
builds.should(QueryBuilders.matchQuery(paramName, value));
}
boolQuery.must(builds);
}else {
QueryBuilder builds = QueryBuilders.termsQuery(paramName, (Collection<?>)paramValue);
boolQuery.must(builds);
}
}else {
// 单个值处理
QueryBuilder builds;
if (analyse) {
builds = QueryBuilders.matchQuery(paramName, paramValue);
} else {
builds = QueryBuilders.termQuery(paramName, paramValue);
}
boolQuery.must(builds);
}
}
// 处理时间范围查询
if (Objects.nonNull(startTime) || Objects.nonNull(endTime)) {
QueryBuilder builds = QueryBuilders.rangeQuery(paramName)
.from(startTime).to(endTime).includeLower(true);
boolQuery.must(builds);
}
}
sourceBuilder.trackTotalHits(true);
sourceBuilder.query(boolQuery);
return sourceBuilder;
}4.1.4 数据删除功能
/**
* 根据文档ID删除数据
*
* @param index 索引名称
* @param documentId 文档ID
*/
public void deleteByDocumentId(String index, String documentId) {
if (!esSwitch) {
return;
}
try {
Request request = new Request("DELETE", "/" + index + "/_doc/" + documentId);
request.addParameters(Collections.<String, String>emptyMap());
Response response = restClient.performRequest(request);
log.info("deleteByDocumentId result : {}", response.getStatusLine().getReasonPhrase());
} catch (Exception e) {
log.error("deleteData error", e);
}
}5. 数据传输对象 (DTO)
5.1 查询条件DTO (EsDataQueryDto)
package com.taopiaopiao.dto;
import lombok.Data;
import java.util.Date;
/**
* @program: 高度还原淘票票网高并发实战项目
* @description: elasticsearch查询参数
* @author: GGBOND
**/
@Data
public class EsDataQueryDto {
/**
* 字段名
* */
private String paramName;
/**
* 字段值
* */
private Object paramValue;
/**
* 如果是时间类型，则不使用paramValue 使用startTime和endTime
* */
private Date startTime;
/**
* 如果是时间类型，则不使用paramValue 使用startTime和endTime
* */
private Date endTime;
/**
* 是否分词查询(默认不分词)
* */
private boolean analyse = false;
}5.2 文档映射DTO (EsDocumentMappingDto)
package com.taopiaopiao.dto;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
/**
* @program: 高度还原淘票票网高并发实战项目
* @description: elasticsearch文档映射
* @author: GGBOND
**/
@Data
@AllArgsConstructor
@NoArgsConstructor
public class EsDocumentMappingDto {
/**
* 字段名
* */
private String paramName;
/**
* 字段类型
* */
private String paramType;
}6. 配置使用示例
6.1 application.yml配置示例
elasticsearch:
ip: 127.0.0.1:9200,127.0.0.1:9201 # Elasticsearch集群地址
userName: elastic # 用户名
passWord: changeme # 密码
esSwitch: true # 是否启用Elasticsearch
esTypeSwitch: false # 是否使用类型（兼容旧版本）
connectTimeOut: 40000 # 连接超时时间
socketTimeOut: 40000 # Socket超时时间
connectionRequestTimeOut: 40000 # 连接请求超时时间
maxConnectNum: 400 # 最大连接数6.2 业务代码使用示例
@Service
public class MovieSearchService {
@Autowired
private BusinessEsHandle businessEsHandle;
// 创建电影索引
public void createMovieIndex() throws IOException {
List<EsDocumentMappingDto> mappings = new ArrayList<>();
mappings.add(new EsDocumentMappingDto("id", "keyword"));
mappings.add(new EsDocumentMappingDto("name", "text"));
mappings.add(new EsDocumentMappingDto("director", "text"));
mappings.add(new EsDocumentMappingDto("actors", "text"));
mappings.add(new EsDocumentMappingDto("types", "keyword"));
mappings.add(new EsDocumentMappingDto("releaseDate", "date"));
mappings.add(new EsDocumentMappingDto("averageScore", "float"));
businessEsHandle.createIndex("movie_index", "movie", mappings);
}
// 添加电影数据
public boolean addMovie(Movie movie) {
Map<String, Object> params = new HashMap<>();
params.put("id", movie.getId());
params.put("name", movie.getName());
params.put("director", movie.getDirector());
params.put("actors", movie.getActors());
params.put("types", movie.getTypes());
params.put("releaseDate", movie.getReleaseDate());
params.put("averageScore", movie.getAverageScore());
return businessEsHandle.add("movie_index", "movie", params, movie.getId());
}
// 搜索电影
public PageInfo<Movie> searchMovies(String keyword, List<String> types, Integer pageNo, Integer pageSize) throws IOException {
List<EsDataQueryDto> queryDtos = new ArrayList<>();
// 关键词搜索（分词）
if (StringUtil.isNotEmpty(keyword)) {
EsDataQueryDto keywordDto = new EsDataQueryDto();
keywordDto.setParamName("name");
keywordDto.setParamValue(keyword);
keywordDto.setAnalyse(true); // 分词查询
queryDtos.add(keywordDto);
}
// 类型过滤
if (CollectionUtil.isNotEmpty(types)) {
EsDataQueryDto typeDto = new EsDataQueryDto();
typeDto.setParamName("types");
typeDto.setParamValue(types);
queryDtos.add(typeDto);
}
// 按评分降序排序
return businessEsHandle.queryPage("movie_index", "movie", null,
queryDtos, "averageScore", null, SortOrder.DESC, pageNo, pageSize, Movie.class);
}
}7. 技术特点与优势
- 统一封装：通过BusinessEsHandle类统一封装Elasticsearch操作，简化业务代码调用
- 灵活配置：支持通过配置文件灵活配置Elasticsearch参数，包括集群地址、认证信息、超时时间等
- 开关机制：提供esSwitch开关，可以方便地控制是否启用Elasticsearch功能
- 兼容性：支持esTypeSwitch参数，兼容不同版本的Elasticsearch（6.x支持类型，7.x不支持类型）
- 丰富功能：支持索引管理、数据CRUD、复杂查询、分页、排序、地理位置查询等功能
- 日志记录：关键操作都有详细的日志记录，便于问题排查
8. 注意事项
- 使用前需确保Elasticsearch服务已启动并可访问
- 合理设置超时时间和连接数，避免资源浪费
- 对于大数据量的查询，建议使用分页查询并合理设置每页大小
- 对于需要分词的字段，设置analyse=true以启用分词查询
- 地理位置查询和排序功能需要正确配置地理坐标字段
- 生产环境建议启用认证并设置安全的用户名密码