ElasticSearch - Fetch more than 10,000 results

Elastic has a default limit of 10,000 result items per search. How can I fetch all 200,000+ items in the index?

Using below throws an error, because Size is larger than 10,000:

 var searchResult = new ElasticSearchResponse<ProductDocument>(_searchClientService
            .Search<ProductDocument>(new CultureInfo(CultureConstants.EnUsCultureName), selector => selector
                .Size(10001).QueryWithPermission(queryContainerDescriptor => BuildQuery(queryContainerDescriptor, searchQuery))));

Reading Elastic documentation using Scroll_id seems to be the way to do it. Is it possible to use scroll_id with Litium.Search.SearchClientService? I can find the scroll_id value but not how to use it. Any example code I can look at?

Litium version: [7.4.3-patch-2007111155]

Yes, it should be possible to use, the SearchClientService are exposing the NEST library function for search and you have some documentation on https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/scrolling-documents.html that describe how it should work with the scrolling functionality.

The example in the NEST documentation uses the ElasticClient.Scroll() method but I cannot find where this is exposed in Litium.Search.SearchClientService. Only Search() and IsConfigured() methods are exposed.

Creating a new ElasticClient() exposes the Scroll() method.

It’s possible to inject the IElasticClient directly, the SearchClientService is wrapping that and setting the index to correct name automatic. Usage of IElasticClient everything need to be done manually.

Untested code but something like this

using System;
using System.Globalization;
using System.Linq;
using Litium.Accelerator.Search;
using Litium.Search;
using Nest;

namespace Litium.Accelerator.Searching
{
    public class TestSearch
    {
        private readonly SearchClientService _searchClientService;
        private readonly IElasticClient _elasticClient;

        public TestSearch(
            SearchClientService searchClientService,
            IElasticClient elasticClient)
        {
            _searchClientService = searchClientService;
            _elasticClient = elasticClient;
        }

        public void Search()
        {
            var scrollTime = new Time(TimeSpan.FromMinutes(5));
            var searchResponse = _searchClientService
                    .Search<ProductDocument>(CultureInfo.CurrentUICulture, selector => selector
                        .Scroll(scrollTime));

            while (searchResponse.Documents.Any())
            {
                ProcessResponse(searchResponse);
                searchResponse = _elasticClient.Scroll<ProductDocument>(scrollTime, searchResponse.ScrollId);
            }
        }

        private void ProcessResponse(ISearchResponse<ProductDocument> searchResponse)
        {

        }
    }
}

Thank you Patric, I got it to work by injecting IElasticClient and using that client for the scroll calls.

Example of working code:

 public List<ProductDocument> ElasticSearchProducts(SearchQuery searchQuery)
        {
            var scrollTime = new Time(TimeSpan.FromMinutes(5));
            var searchResult = new ElasticSearchResponse<ProductDocument>(_searchClientService
                .Search<ProductDocument>(new CultureInfo(CultureConstants.EnUsCultureName), selector => selector
                    .Size(10000).Scroll(scrollTime).QueryWithPermission(queryContainerDescriptor => BuildQuery(queryContainerDescriptor, searchQuery))));

            var productHits = new List<ProductDocument>();

            while (searchResult.Response.Documents.Any())
            {
                productHits.AddRange(searchResult.Response?.Documents?.ToList());
                searchResult = new ElasticSearchResponse<ProductDocument>(_elasticClient.Scroll<ProductDocument>(scrollTime, searchResult.Response.ScrollId));
            }

            return productHits;
        }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.