add emoji search

[strong-xiaoge]

Hope to increase support for emoji search

[sanikolaev]

Here's an example:

mysql> drop table if exists t; create table t(f text) charset_table='non_cjk,U+1F601'; call keywords('abc😁', 't'); insert into t(f) values('abc'),('abc😁'); select * from t where match('abc'); select * from t where match('abc😁');
--------------
drop table if exists t
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
create table t(f text) charset_table='non_cjk,U+1F601'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
call keywords('abc😁', 't')
--------------

+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | abc😁       | abc😁        |
+------+-----------+------------+
1 row in set (0.00 sec)

--------------
insert into t(f) values('abc'),('abc😁')
--------------

Query OK, 2 rows affected (0.00 sec)

--------------
select * from t where match('abc')
--------------

+---------------------+------+
| id                  | f    |
+---------------------+------+
| 1515816049293918213 | abc  |
+---------------------+------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

--------------
select * from t where match('abc😁')
--------------

+---------------------+---------+
| id                  | f       |
+---------------------+---------+
| 1515816049293918214 | abc😁     |
+---------------------+---------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

[strong-xiaoge]

At the same time, it also needs to support the use of Chinese search, how to set it?

[sanikolaev]

- charset_table='non_cjk,U+1F601'
+ charset_table='chinese,non_cjk,U+1F601'
+ morphology='icu_chinese'

Please take this course for more details - https://play.manticoresearch.com/icu-chinese/

[strong-xiaoge]

I understand. Thank you for your answer.

[kissss]

@strong-xiaoge 你们现在有在使用吗，我们也正在调研使用，能否加个联系交流请教一下

[strong-xiaoge]

有使用

[sanikolaev]

@kissss @strong-xiaoge Be aware, there's another way of Chinese segmentation via the Jieba library now - https://manual.manticoresearch.com/dev/Creating_a_table/NLP_and_tokenization/Languages_with_continuous_scripts#Chinese,-Japanese-and-Korean-(CJK)-and-Thai-languages

It's available in the dev packages - https://mnt.cr/dev/nightly

[strong-xiaoge]

@kissss @strong-xiaoge Be aware, there's another way of Chinese segmentation via the Jieba library now - https://manual.manticoresearch.com/dev/Creating_a_table/NLP_and_tokenization/Languages_with_continuous_scripts#Chinese,-Japanese-and-Korean-(CJK)-and-Thai-languages

It's available in the dev packages - https://mnt.cr/dev/nightly

Does it work better

[sanikolaev]

According to our tests, it gives better segmentation quality than ICU. It also makes it easier to use custom dictionaries - https://manual.manticoresearch.com/dev/Creating_a_table/NLP_and_tokenization/Morphology#jieba_user_dict_path

We'd love to get feedback from native speakers.