Rebuilding the elasticsearch index one at the time?

Questions
1

So, when we rebuild the entrys in elasticindex our site is basically unusable for 30min because everything disappears and then slowly gets added back in.

Im thinking of fixing this myself but just wanted to check since we cant be the only one with this problem, are a soulution in the works?

If not, can we at least have a additional trigger link in admin, to rebuild insted of replacing? Since I like them both :slight_smile:

Also, when it comse to index versions, a dropdown of what prefix to rebuild and what prefix to use for Querys whould save us when we need to change the schema, (I dont know how to fix this myselfe)

Litium version: 7.3

2

When the Elasticsearch connection was built we tried to include blue/green rebuilding of indices to allow the index to be online during the time the index still was under index creation. But due to time we was cutting this from the scope.

When rebuild and not remove the previous record it may always be record left in the search index that are orphaned (usually the reason to rebuild the index) was left in the index and still returned as hits when doing searches.

Here we have some questions that I can think of right now regarding this

  1. When processing the index queue, should the document be written to both the old and new version of the index.
  2. When should a new version of the index be created, always or with a special button.
  3. When the new version should be created, should that include all document types (products, categories, pages) or separately per document type.
  4. When verifying the index schema (field with their types) and the new version of the schema not can be applied, should the system automatic schedule a rebuild of the index for that document type.
  5. When a new schema can be applied for a document type, should the system automatic trigger an update of all items for that document type.
3

I haven’t test it my self but I think you can add a decorator to add an “Update” function, maybe something like this

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Litium.Runtime.DependencyInjection;
using Litium.Search;
using Litium.Search.Indexing;

namespace Litium.Accelerator
{
    [ServiceDecorator(typeof(IndexConfigurationService))]
    internal class IndexConfigurationServiceDecorator : IndexConfigurationService
    {
        private readonly IndexConfigurationService _parent;
        private readonly IndexQueueService _indexQueueService;
        private readonly Dictionary<string, IIndexMappingConfiguration> _indexes = new Dictionary<string, IIndexMappingConfiguration>(StringComparer.OrdinalIgnoreCase);

        public IndexConfigurationServiceDecorator(
            IndexConfigurationService parent,
            IEnumerable<IIndexMappingConfiguration> indexConfigurations,
            IndexQueueService indexQueueService)
        {
            _parent = parent;
            _indexQueueService = indexQueueService;

            foreach (var config in indexConfigurations)
            {
                _indexes.Add(config.IndexName, config);
            }
        }

        public override IndexConfigurationSummary Get(string index)
        {
            var item = _parent.Get(index);
            item.Actions.Add(new IndexConfigurationSummary.IndexAction { Enabled = true, Name = "UpdateIndex" });
            return item;
        }

        public override IEnumerable<IndexConfigurationSummary> GetAll()
        {
            foreach (var item in _parent.GetAll())
            {
                item.Actions = item.Actions.ToList();
                item.Actions.Add(new IndexConfigurationSummary.IndexAction { Enabled = true, Name = "UpdateIndex" });
                yield return item;
            }
        }

        public override async Task<IndexConfigurationActionResult> InvokeActionAsync(string index, string action, object args = null)
        {
            if (action == "UpdateIndex")
            {
                var config = _indexes.TryGetValue(index, out var value)
                   ? value
                   : throw new ArgumentException("Index does not exists.", nameof(index));

                return await QueueIndexUpdateAsync(config);
            }

            return await _parent.InvokeActionAsync(index, action, args);
        }

        private async Task<IndexConfigurationActionResult> QueueIndexUpdateAsync(IIndexMappingConfiguration config)
        {
            return await config.QueueIndexRebuildAsync(_indexQueueService);
        }
    }
}
1 Like
4

Thanks! Is there also a Decorator I can use to adjust the prefix or index names to use when building or connecting? So that we can create a simple green/blue to start of with?

5

No, the prefix are added in multiple places without the possibility to change with a decorator.

6

For those of you who have the opportunity, you are welcome to answer the numbered list of questions I have posed above. This will help in the development of this functionality.

7

This is what make sense for our specific use case:

  1. Incomning changes need to be added to booth queues, but this is only needed when in the process of switching index/rebuilding untill the new one is active.
    A toggle to show advanced option that reveales a active tickbox next to each index would be nice.

Or, just automaticly activate the index when a rebuild is started and deactivate it the first time its set as the primary.

  1. Button, 99% of the time we have no schema change and can reuse the blue/green for a long time.

  2. a version/schema should be same for all index, if we change the schema for pages, rebuilding the products whould be fine to make the naming consistent when debugging.

  3. Is fine just to trow if its a missmatch, we feel like editing schema we all undertand that we need to rebuild befor going live with the release.

  4. It dose not matter, update or creating a new

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.