Track token scores #571

KarelVesely84 · 2024-02-02T15:34:15Z

Hello,
I modified the code, so that the per-token scores are accessible via python API
(ys_probs, lm_probs, context_scores).

This is tracked during the modified beam search decoding of online transduces,
and it is exported for the BestHypothesis.

Would you be interesting in having this functionality in the codebase ?
This can be used as confidences from the client side...

But I did not evaluate it yet, to see how "useful" these numbers are.
I am ready to do changes as well. I'd like to open the discussion...

Best regards
K. Veselý

csukuangfj · 2024-02-03T06:49:17Z

Would you be interesting in having this functionality in the codebase ?

Yes, it is a useful feature and will benefit some users who want to get confidence for each token.

In addition to the modified_beam_search, could you also update the greedy_search?

KarelVesely84 · 2024-02-14T16:27:43Z

Okay, i extended it also for the greedy_search.
I did not test it yet... I'll do it soon...

KarelVesely84 · 2024-02-15T17:27:51Z

Hi Fangyun @csukuangfj ,
now, the per-token confidences are accessible both for greedy_search and modified_beam_search
of transducer decoding.

I manually tested it with CPU compiled sherpa-onnx through python API.

In the GH test-workflow I see some segmentation fault for Online-CTC in linux-gpu test.
Is this because the OnlineRecognizerResult class was extended in my PR ?

Do you also run the tests locally to investigate ?
K.

csukuangfj · 2024-02-17T04:45:32Z

Thanks! I will have time after today. Will look into it tomorrow.

csukuangfj · 2024-02-20T12:41:38Z

I see the error.

It is a use-after-free error.

Please see my comments about the code

csukuangfj · 2024-02-20T12:42:05Z

sherpa-onnx/csrc/online-recognizer.cc

-
+/// Helper for `OnlineRecognizerResult::AsJsonString()`
+template<typename T>
+const std::string& VecToString(const std::vector<T>& vec, int32_t precision = 6) {


Suggested change

const std::string& VecToString(const std::vector<T>& vec, int32_t precision = 6) {

std::string VecToString(const std::vector<T>& vec, int32_t precision = 6) {

csukuangfj · 2024-02-20T12:42:30Z

sherpa-onnx/csrc/online-recognizer.cc

+
+/// Helper for `OnlineRecognizerResult::AsJsonString()`
+template<>  // explicit specialization for T = std::string
+const std::string& VecToString<std::string>(const std::vector<std::string>& vec,


Suggested change

const std::string& VecToString<std::string>(const std::vector<std::string>& vec,

std::string VecToString<std::string>(const std::vector<std::string>& vec,

Please never return a temporary reference from a function.

Aha, thank you.

My bad. I remember in some context returning a local object as a const reference worked, but here I did it badly.
https://stackoverflow.com/questions/13318257/const-reference-to-temporary-vs-return-value-optimization

Best, K.

I remember in some context returning a local object as a const reference worked

Yes, you are right. But in this case, the temporary reference is from the function return value of a temporary object.
Think twice, there are two indirections.

Suppose obj is a local variable in a function. It is totally fine to call

return obj

to return a const reference.

But it is not valid to invoke

return obj.someMethod();

since obj is destroyed when the function returns.

Yes, you are right.

The code is fixed now. I tested the JSON output from python, and it looks fine.

Also now the lm_probs and contexts scores are filled only if LM and ContextGraph are used.
Otherwise there are empty lists to save bandwidth.

Thank you for finding the bug,
Karel

csukuangfj · 2024-02-27T06:19:00Z

I see that you have removed WIP. Do you think it is ready for review and merge?

- for best path of the modified-beam-search decoding of transducer

…1 API of OnlineRecognitionResult

- export un-scaled lm_probs (modified-beam search, online-transducer) - polishing

KarelVesely84 · 2024-02-27T13:19:57Z

Yes, it is almost ready to be merged.
Just making sure that the linting check passes...

It is ready for the review.

KarelVesely84 · 2024-02-28T10:00:23Z

Ok, the workflow tests seem fine, including the linter style-check... It is ready...

csukuangfj · 2024-02-28T10:02:35Z

I will have a second look after dinner. Thanks!

csukuangfj

Thanks again!

Left some minor comments. Otherwise, it looks good to me.

csukuangfj · 2024-02-28T11:35:09Z

sherpa-onnx/csrc/online-recognizer.cc

+  std::ostringstream oss;
+  oss << "[ ";
+  std::string sep = "";
+  for (auto item : vec) {


Suggested change

for (auto item : vec) {

for (const auto i&tem : vec) {

csukuangfj · 2024-02-28T11:37:41Z

sherpa-onnx/csrc/online-transducer-greedy-search-decoder.cc

+        LogSoftmax(p_logit, vocab_size);  // renormalize probabilities,
+                                          // save time by doing it only for
+                                          // emitted symbols
+        float *p_logprob = p_logit;  // rename p_logit as p_logprob,


Suggested change

float *p_logprob = p_logit; // rename p_logit as p_logprob,

const float *p_logprob = p_logit; // rename p_logit as p_logprob,

… Result smaller)

KarelVesely84 · 2024-02-28T16:11:59Z

okay, the two suggestions are implemented.... thank you...

csukuangfj · 2024-02-28T22:27:34Z

Thanks!

* add export of per-token scores (ys, lm, context) - for best path of the modified-beam-search decoding of transducer * refactoring JSON export of OnlineRecognitionResult, extending pybind11 API of OnlineRecognitionResult * export per-token scores also for greedy-search (online-transducer) - export un-scaled lm_probs (modified-beam search, online-transducer) - polishing * fill lm_probs/context_scores only if LM/ContextGraph is present (make Result smaller)

KarelVesely84 force-pushed the track_token_scores branch from f8d2abd to 6c5d358 Compare February 14, 2024 15:26

KarelVesely84 changed the title ~~[not-for-merge] Track token scores~~ [WIP] Track token scores Feb 14, 2024

KarelVesely84 force-pushed the track_token_scores branch from 6c5d358 to a4bc688 Compare February 15, 2024 16:39

csukuangfj reviewed Feb 20, 2024

View reviewed changes

KarelVesely84 force-pushed the track_token_scores branch 4 times, most recently from 0ab7a40 to e442564 Compare February 27, 2024 06:06

KarelVesely84 changed the title ~~[WIP] Track token scores~~ Track token scores Feb 27, 2024

KarelVesely84 mentioned this pull request Feb 27, 2024

Confidence scores with Zipformer models #490

Open

KarelVesely84 force-pushed the track_token_scores branch from e442564 to a164e13 Compare February 27, 2024 13:09

KarelVesely84 added 3 commits February 27, 2024 14:11

add export of per-token scores (ys, lm, context)

0f2b73d

- for best path of the modified-beam-search decoding of transducer

refactoring JSON export of OnlineRecognitionResult, extending pybind1…

570bc29

…1 API of OnlineRecognitionResult

export per-token scores also for greedy-search (online-transducer)

c1569f7

- export un-scaled lm_probs (modified-beam search, online-transducer) - polishing

KarelVesely84 force-pushed the track_token_scores branch 2 times, most recently from 48e4a6a to 4955fe8 Compare February 27, 2024 13:15

csukuangfj approved these changes Feb 28, 2024

View reviewed changes

fill lm_probs/context_scores only if LM/ContextGraph is present (make…

2cba75f

… Result smaller)

KarelVesely84 force-pushed the track_token_scores branch from 4955fe8 to 2cba75f Compare February 28, 2024 16:09

csukuangfj merged commit 38c072d into k2-fsa:master Feb 28, 2024
64 of 227 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track token scores #571

Track token scores #571

KarelVesely84 commented Feb 2, 2024

csukuangfj commented Feb 3, 2024

KarelVesely84 commented Feb 14, 2024

KarelVesely84 commented Feb 15, 2024 •

edited

Loading

csukuangfj commented Feb 17, 2024

csukuangfj commented Feb 20, 2024

csukuangfj Feb 20, 2024

csukuangfj Feb 20, 2024

csukuangfj Feb 20, 2024

KarelVesely84 Feb 26, 2024

csukuangfj Feb 26, 2024

KarelVesely84 Feb 26, 2024

csukuangfj commented Feb 27, 2024

KarelVesely84 commented Feb 27, 2024 •

edited

Loading

KarelVesely84 commented Feb 28, 2024

csukuangfj commented Feb 28, 2024

csukuangfj left a comment

csukuangfj Feb 28, 2024

csukuangfj Feb 28, 2024

KarelVesely84 commented Feb 28, 2024

csukuangfj commented Feb 28, 2024

	const std::string& VecToString(const std::vector<T>& vec, int32_t precision = 6) {
	std::string VecToString(const std::vector<T>& vec, int32_t precision = 6) {

	const std::string& VecToString<std::string>(const std::vector<std::string>& vec,
	std::string VecToString<std::string>(const std::vector<std::string>& vec,

	float *p_logprob = p_logit; // rename p_logit as p_logprob,
	const float *p_logprob = p_logit; // rename p_logit as p_logprob,

Track token scores #571

Track token scores #571

Conversation

KarelVesely84 commented Feb 2, 2024

csukuangfj commented Feb 3, 2024

KarelVesely84 commented Feb 14, 2024

KarelVesely84 commented Feb 15, 2024 • edited Loading

csukuangfj commented Feb 17, 2024

csukuangfj commented Feb 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csukuangfj commented Feb 27, 2024

KarelVesely84 commented Feb 27, 2024 • edited Loading

KarelVesely84 commented Feb 28, 2024

csukuangfj commented Feb 28, 2024

csukuangfj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KarelVesely84 commented Feb 28, 2024

csukuangfj commented Feb 28, 2024

KarelVesely84 commented Feb 15, 2024 •

edited

Loading

KarelVesely84 commented Feb 27, 2024 •

edited

Loading