- Create the Index with Query-Time Analyzer Only:
<code>PUT /local_persons
{
"settings": {
"analysis": {
"analyzer": {
"person_search_analyzer": {
"type": "custom",
"char_filter": ["remove_special_chars"],
"filter": ["lowercase"],
"tokenizer": "whitespace"
}
},
"char_filter": {
"remove_special_chars": {
"type": "pattern_replace",
"pattern": "[^a-zA-Z0-9]",
"replacement": ""
}
}
}
}
}
</code>
<code>PUT /local_persons
{
"settings": {
"analysis": {
"analyzer": {
"person_search_analyzer": {
"type": "custom",
"char_filter": ["remove_special_chars"],
"filter": ["lowercase"],
"tokenizer": "whitespace"
}
},
"char_filter": {
"remove_special_chars": {
"type": "pattern_replace",
"pattern": "[^a-zA-Z0-9]",
"replacement": ""
}
}
}
}
}
</code>
PUT /local_persons
{
"settings": {
"analysis": {
"analyzer": {
"person_search_analyzer": {
"type": "custom",
"char_filter": ["remove_special_chars"],
"filter": ["lowercase"],
"tokenizer": "whitespace"
}
},
"char_filter": {
"remove_special_chars": {
"type": "pattern_replace",
"pattern": "[^a-zA-Z0-9]",
"replacement": ""
}
}
}
}
}
- Indexed the data with special characters:
<code>PUT /local_persons/_doc/1
{
"id": 1,
"firstName": "Re'mo",
"lastName": "D'souza",
"email": "[email protected],
"dateOfBirth": "1973-01-01",
"isActive": 1
}
</code>
<code>PUT /local_persons/_doc/1
{
"id": 1,
"firstName": "Re'mo",
"lastName": "D'souza",
"email": "[email protected],
"dateOfBirth": "1973-01-01",
"isActive": 1
}
</code>
PUT /local_persons/_doc/1
{
"id": 1,
"firstName": "Re'mo",
"lastName": "D'souza",
"email": "[email protected],
"dateOfBirth": "1973-01-01",
"isActive": 1
}
Now searching for the person at the query time:
Approach-1: Search using query_string (with analyzer at query time)
<code>GET /local_persons/_search
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "remo",
"fields": ["firstName"],
"analyzer": "person_search_analyzer"
}
},
{
"query_string": {
"query": "dsouza",
"fields": ["lastName"],
"analyzer": "person_search_analyzer"
}
}
]
}
}
}
</code>
<code>GET /local_persons/_search
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "remo",
"fields": ["firstName"],
"analyzer": "person_search_analyzer"
}
},
{
"query_string": {
"query": "dsouza",
"fields": ["lastName"],
"analyzer": "person_search_analyzer"
}
}
]
}
}
}
</code>
GET /local_persons/_search
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "remo",
"fields": ["firstName"],
"analyzer": "person_search_analyzer"
}
},
{
"query_string": {
"query": "dsouza",
"fields": ["lastName"],
"analyzer": "person_search_analyzer"
}
}
]
}
}
}
Approach 2: Using the match query
<code>GET /local_persons/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"firstName": {
"query": "remo",
"analyzer": "person_search_analyzer"
}
}
},
{
"match": {
"lastName": {
"query": "dsouza",
"analyzer": "person_search_analyzer"
}
}
}
]
}
}
}
</code>
<code>GET /local_persons/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"firstName": {
"query": "remo",
"analyzer": "person_search_analyzer"
}
}
},
{
"match": {
"lastName": {
"query": "dsouza",
"analyzer": "person_search_analyzer"
}
}
}
]
}
}
}
</code>
GET /local_persons/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"firstName": {
"query": "remo",
"analyzer": "person_search_analyzer"
}
}
},
{
"match": {
"lastName": {
"query": "dsouza",
"analyzer": "person_search_analyzer"
}
}
}
]
}
}
}
But both of the approaches above are giving empty results.
Elastic is not returning response because it has consider ‘standard’ as default analyzer for indexing as you didn’t defined index mapping and analyzer.
You can execute below API to get info about what is indexed in ES:
<code>POST local_persons/_analyze
{
"text":["Re'mo"],
"analyzer" : "standard"
}
</code>
<code>POST local_persons/_analyze
{
"text":["Re'mo"],
"analyzer" : "standard"
}
</code>
POST local_persons/_analyze
{
"text":["Re'mo"],
"analyzer" : "standard"
}
Response:
<code>{
"tokens": [
{
"token": "re'mo",
"start_offset": 0,
"end_offset": 5,
"type": "<ALPHANUM>",
"position": 0
}
]
}
</code>
<code>{
"tokens": [
{
"token": "re'mo",
"start_offset": 0,
"end_offset": 5,
"type": "<ALPHANUM>",
"position": 0
}
]
}
</code>
{
"tokens": [
{
"token": "re'mo",
"start_offset": 0,
"end_offset": 5,
"type": "<ALPHANUM>",
"position": 0
}
]
}
Lets see result of your analyzer:
<code>POST local_persons/_analyze
{
"text":["Re'mo"],
"analyzer" : "person_search_analyzer"
}
</code>
<code>POST local_persons/_analyze
{
"text":["Re'mo"],
"analyzer" : "person_search_analyzer"
}
</code>
POST local_persons/_analyze
{
"text":["Re'mo"],
"analyzer" : "person_search_analyzer"
}
Response:
<code>{
"tokens": [
{
"token": "remo",
"start_offset": 0,
"end_offset": 5,
"type": "word",
"position": 0
}
]
}
</code>
<code>{
"tokens": [
{
"token": "remo",
"start_offset": 0,
"end_offset": 5,
"type": "word",
"position": 0
}
]
}
</code>
{
"tokens": [
{
"token": "remo",
"start_offset": 0,
"end_offset": 5,
"type": "word",
"position": 0
}
]
}
If you observe both the response then you can clearly see that both are giving different response and hence Elastic is not returning response.
So, you can define index mapping like below which will make sure that index and search time it will use same analyzer and will return expected result as well.
<code>PUT /local_persons1
{
"settings": {
"analysis": {
"analyzer": {
"person_search_analyzer": {
"type": "custom",
"char_filter": ["remove_special_chars"],
"filter": ["lowercase"],
"tokenizer": "whitespace"
}
},
"char_filter": {
"remove_special_chars": {
"type": "pattern_replace",
"pattern": "[^a-zA-Z0-9]",
"replacement": ""
}
}
}
},
"mappings": {
"properties": {
"firstName":{
"type": "text",
"analyzer": "person_search_analyzer"
},
"lastName":{
"type": "text",
"analyzer": "person_search_analyzer"
}
}
}
}
</code>
<code>PUT /local_persons1
{
"settings": {
"analysis": {
"analyzer": {
"person_search_analyzer": {
"type": "custom",
"char_filter": ["remove_special_chars"],
"filter": ["lowercase"],
"tokenizer": "whitespace"
}
},
"char_filter": {
"remove_special_chars": {
"type": "pattern_replace",
"pattern": "[^a-zA-Z0-9]",
"replacement": ""
}
}
}
},
"mappings": {
"properties": {
"firstName":{
"type": "text",
"analyzer": "person_search_analyzer"
},
"lastName":{
"type": "text",
"analyzer": "person_search_analyzer"
}
}
}
}
</code>
PUT /local_persons1
{
"settings": {
"analysis": {
"analyzer": {
"person_search_analyzer": {
"type": "custom",
"char_filter": ["remove_special_chars"],
"filter": ["lowercase"],
"tokenizer": "whitespace"
}
},
"char_filter": {
"remove_special_chars": {
"type": "pattern_replace",
"pattern": "[^a-zA-Z0-9]",
"replacement": ""
}
}
}
},
"mappings": {
"properties": {
"firstName":{
"type": "text",
"analyzer": "person_search_analyzer"
},
"lastName":{
"type": "text",
"analyzer": "person_search_analyzer"
}
}
}
}
Now both of your query will return the response even if you don’t provide analyzer in response.
Query:
<code>GET /local_persons1/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"firstName": {
"query": "remo"
}
}
},
{
"match": {
"lastName": {
"query": "dsouza"
}
}
}
]
}
}
}
</code>
<code>GET /local_persons1/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"firstName": {
"query": "remo"
}
}
},
{
"match": {
"lastName": {
"query": "dsouza"
}
}
}
]
}
}
}
</code>
GET /local_persons1/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"firstName": {
"query": "remo"
}
}
},
{
"match": {
"lastName": {
"query": "dsouza"
}
}
}
]
}
}
}
Response:
<code>{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.5753642,
"hits": [
{
"_index": "local_persons1",
"_id": "1",
"_score": 0.5753642,
"_source": {
"id": 1,
"firstName": "Re'mo",
"lastName": "D'souza",
"email": "[email protected]",
"dateOfBirth": "1973-01-01",
"isActive": 1
}
}
]
}
</code>
<code>{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.5753642,
"hits": [
{
"_index": "local_persons1",
"_id": "1",
"_score": 0.5753642,
"_source": {
"id": 1,
"firstName": "Re'mo",
"lastName": "D'souza",
"email": "[email protected]",
"dateOfBirth": "1973-01-01",
"isActive": 1
}
}
]
}
</code>
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.5753642,
"hits": [
{
"_index": "local_persons1",
"_id": "1",
"_score": 0.5753642,
"_source": {
"id": 1,
"firstName": "Re'mo",
"lastName": "D'souza",
"email": "[email protected]",
"dateOfBirth": "1973-01-01",
"isActive": 1
}
}
]
}
2