Pagination to have more than 20 elements in the response, after a search: mastodon Client.search().search Content(keyword).execute()

[juniortestjr]

Thank you for developing the successor to the Mastodon project. I'm writing this message because I'm working on a project and I'm using this library to search for messages based on a keyword. Here is a part from my code :

ArrayList<String> html_statuses_list = new ArrayList<>();
Search search_results = mastodonClient.search().searchContent(keyword).execute();
search_results.getStatuses().forEach(status -> html_statuses_list.add(status.getContent()))

the program works, but I receive a maximum of 20 elements (messages) each time. Apparently, normally there are more elements in response and maybe it's because there is pagination that I don't get more than 20 elements.

If so, I wonder how we can adjust the pagination?

[bocops]

The searchContent() method has an optional offset parameter that can be used to skip some of the results. You can use that for pagination.

That said, I'm not sure if that method shouldn't use our Range class to combine parameters like maxId, minId, limit instead of exposing them separately, and whether that endpoint supports proper pagination via a Link header as outlined in https://docs.joinmastodon.org/api/guidelines/#pagination.

[PattaFeuFeu]

That said, I'm not sure if that method shouldn't use our Range class to combine parameters like maxId, minId, limit instead of exposing them separately

I was just thinking the same but the documentation doesn’t mention since_id or pagination via Link header so I’m not sure if we should. This seems to be different. 😕

[bocops]

Yeah, search_controller.rb does not currently have an after_action that inserts pagination headers, so we're on our own regarding pagination.

I think we could still use Range without having a defined since_id, as long as that parameter would just be unused in the method. If we wanted, we could also offer some extension or other "helper" method that generated a valid "next" range from the returned Search by just looking for the maximum ID in that result set and using it (+1?) as minId of the returned Range.

EDIT: It is a bit unclear what that would do for mixed result sets (e.g. containing both accounts and hashtags), but it might work for single-type searches.

[PattaFeuFeu]

I think we could still use Range without having a defined since_id

Mhm, I don’t think we should expose something and then not use parts of it. 😕

It is a bit unclear what that would do for mixed result sets (e.g. containing both accounts and hashtags), but it might work for single-type searches.

That’s probably the main reason why pagination is not currently available for that endpoint.

Are ids always ensured to be sorted descending or ascending so that we could even use minId and maxId to get what we want?

[bocops]

Judging by the examples in Mastodon documentation, it seems as if this might even be handled differently depending on type:

• accounts looks as if it is sorted by ascending ID
• statuses looks as if it is sorted by descending createdAt (which probably also means descending ID)
• hashtags doesn't even return IDs (because those are only exposed as part of Admin::Tag?)

In all cases, the result set is too small to really be certain about anything - but it begs the question what the search endpoint would even return when, for example, not specifying any type but a minId.

[PattaFeuFeu]

Hi @juniortestjr! Thank you for using BigBone. 😊

You can find the documentation and source code of that method in SearchMethods.kt:

bigbone/bigbone/src/main/kotlin/social/bigbone/api/method/SearchMethods.kt

Lines 31 to 63 in 12d6648

	/**
	* Search for content in accounts, statuses and hashtags.
	* @param query string to search for
	* @param type to specify if you are looking for a specific typology
	* @param resolve whether to resolve non-local accounts
	* @param following whether to include accounts that the user is following
	* @param excludeUnreviewed whether to look for trending tags
	* @param accountId to only return statuses authored by the user account
	* @param maxId to return results whose id will be lesser than this one
	* @param minId to return results whose id will be newer than this one
	* @param limit to limit number of results within a range of 20-40 per category
	* @param offset to skip the first n results
	* @see <a href="https://docs.joinmastodon.org/methods/search/">Mastodon API documentation: methods/search</a>
	*/
	@JvmOverloads
	fun searchContent(
	query: String,
	type: SearchType? = null,
	resolve: Boolean = false,
	following: Boolean = false,
	excludeUnreviewed: Boolean = false,
	accountId: String? = null,
	maxId: String? = null,
	minId: String? = null,
	limit: Int? = null,
	offset: Int? = null
	): MastodonRequest<Search> {
	return client.getMastodonRequest(
	endpoint = "api/v2/search",
	method = MastodonClient.Method.GET,
	parameters = buildParameters(query, type, resolve, following, excludeUnreviewed, accountId, maxId, minId, limit, offset)
	)
	}

In Kotlin, we were able to define default parameters for all values but query, but in Java you don’t have that luxury unfortunately. We do provide overloads so that you have the opportunity to only use some of the values. Unfortunately for you, they are set up in a way where the pagination parameters are at the very end, so you must define all other parameters. 😕

With the rest of your code effectively staying the same, you could do the following:

First 20 results:

final Search searchResult = client.search().searchContent(
                searchTerm,
                null,
                false,
                false,
                false,
                null,
                null,
                null,
                null,
                null
        ).execute();

Next 20 results:

final Search searchResult = client.search().searchContent(
                searchTerm,
                null,
                false,
                false,
                false,
                null,
                null,
                null,
                null,
                20
        ).execute();

(mind the 20 as the last parameter—that’s the offset. Set that to 40 for the next 20, then 60, 80, …)

[PattaFeuFeu]

ArrayList<String> html_statuses_list = new ArrayList<>();
...
search_results.getStatuses().forEach(status -> html_statuses_list.add(status.getContent()))

By the way: search_results.getStatuses() already returns a List, so you don’t necessarily need html_statuses_list at all.
But if you do, you could also just call ArrayList<String> html_statuses_list = new ArrayList<>(search_results.getStatuses());

[juniortestjr]

it's OK ! , thanks to your help and advice, the pagination works. Thank you everyone ! great for your quick responses!