Elastic has a default limit of 10,000 result items per search. How can I fetch all 200,000+ items in the index?
Using below throws an error, because Size is larger than 10,000:
var searchResult = new ElasticSearchResponse<ProductDocument>(_searchClientService
.Search<ProductDocument>(new CultureInfo(CultureConstants.EnUsCultureName), selector => selector
.Size(10001).QueryWithPermission(queryContainerDescriptor => BuildQuery(queryContainerDescriptor, searchQuery))));
Reading Elastic documentation using Scroll_id seems to be the way to do it. Is it possible to use scroll_id with Litium.Search.SearchClientService? I can find the scroll_id value but not how to use it. Any example code I can look at?
The example in the NEST documentation uses the ElasticClient.Scroll() method but I cannot find where this is exposed in Litium.Search.SearchClientService. Only Search() and IsConfigured() methods are exposed.
Creating a new ElasticClient() exposes the Scroll() method.
It’s possible to inject the IElasticClient directly, the SearchClientService is wrapping that and setting the index to correct name automatic. Usage of IElasticClient everything need to be done manually.
Untested code but something like this
using System;
using System.Globalization;
using System.Linq;
using Litium.Accelerator.Search;
using Litium.Search;
using Nest;
namespace Litium.Accelerator.Searching
{
public class TestSearch
{
private readonly SearchClientService _searchClientService;
private readonly IElasticClient _elasticClient;
public TestSearch(
SearchClientService searchClientService,
IElasticClient elasticClient)
{
_searchClientService = searchClientService;
_elasticClient = elasticClient;
}
public void Search()
{
var scrollTime = new Time(TimeSpan.FromMinutes(5));
var searchResponse = _searchClientService
.Search<ProductDocument>(CultureInfo.CurrentUICulture, selector => selector
.Scroll(scrollTime));
while (searchResponse.Documents.Any())
{
ProcessResponse(searchResponse);
searchResponse = _elasticClient.Scroll<ProductDocument>(scrollTime, searchResponse.ScrollId);
}
}
private void ProcessResponse(ISearchResponse<ProductDocument> searchResponse)
{
}
}
}
Thank you Patric, I got it to work by injecting IElasticClient and using that client for the scroll calls.
Example of working code:
public List<ProductDocument> ElasticSearchProducts(SearchQuery searchQuery)
{
var scrollTime = new Time(TimeSpan.FromMinutes(5));
var searchResult = new ElasticSearchResponse<ProductDocument>(_searchClientService
.Search<ProductDocument>(new CultureInfo(CultureConstants.EnUsCultureName), selector => selector
.Size(10000).Scroll(scrollTime).QueryWithPermission(queryContainerDescriptor => BuildQuery(queryContainerDescriptor, searchQuery))));
var productHits = new List<ProductDocument>();
while (searchResult.Response.Documents.Any())
{
productHits.AddRange(searchResult.Response?.Documents?.ToList());
searchResult = new ElasticSearchResponse<ProductDocument>(_elasticClient.Scroll<ProductDocument>(scrollTime, searchResult.Response.ScrollId));
}
return productHits;
}