Elastic ProductDocuments is not removed only updated

[Robin.w]

We have a field on our products that determens if thy should be included in the ElasticIndex, this works fine on initial index, but if we change this after the product is indexed. The Product is not removed, only not updated when we save it.

This code is that run in our ProductIndexDocumentBuilder()

// Remove all documents for all channel-combinations
		foreach (var channel in channels.Where(x =>     !usedChannelSystemIds.Contains(x.SystemId)))
			{
				var cultureInfo = _languageService.Get(channel.ProductLanguageSystemId.GetValueOrDefault()).CultureInfo;
				yield return (cultureInfo, new RemoveDocument(new ProductDocument
				{
					ArticleNumber = baseProduct.Id,
					ChannelSystemId = channel.SystemId
				}));
				foreach (var variant in variants)
				{
					yield return (cultureInfo, new RemoveDocument(new ProductDocument
					{
						ArticleNumber = variant.Id,
						ChannelSystemId = channel.SystemId
					}));
				}
			}

Is this not suppose to remove it from the index? Do we need to take some more action to remove an item from the index? Is this code only relevent if the product it selfe is deleated from litium?

Edit: Do i need to set the item.Action = IndexAction.Delete ? I dont want to always have this behavior, so i dont want to set it in the event listener? Do i have any options?

Litium version: 7.6

[patric.forsgard]

The RemoveDocument should trigger an delete on the index document. Can you check the elasticsearch.log to see what command that is executed?

[Robin.w]

Should it do this regardless of Action?

[NilsN]

foreach (var channel in channels.Where(x => !usedChannelSystemIds.Contains(x.SystemId)))

Doesn’t this mean it will only remove documents for channels that are not used by the products?

[patric.forsgard]

@Robin.w

Should it do this regardless of Action?

Yes, action Delete on the index queue item is only to indicate that we only want to remove the item from index and not build any new document.

[patric.forsgard]

Did you find a solution on your problem?

[NilsN]

I tested this locally by adding a boolean Hide from Elastic on a variant. In BuildIndexDocuments I except all variants that are set to be hidden, before calling CreateModelsPerChannel. Variants are excluded as expected.

When I rebuild the index the value is adhered to and if the value is false, the document is not created. If I change from Yes to No on the variant, it’s added as a document. Below is the bulk sent when going from Yes to No on 1 variant. All 3 variants are expected to be indexed.

POST http://localhost:9200/_bulk {"index":{"_id":"637471676729630133-06160105-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160105-0000-001","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }
{"index":{"_id":"637471676729630133-06160106-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160106-0000-001","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }
{"index":{"_id":"637471676729630133-06160107-0000-001_scbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160107-0000-001_s","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }

Going from No to Yes on 1 variant below. 2 variants are expected to be indexed.

POST http://localhost:9200/_bulk {"index":{"_id":"637471676729630133-06160105-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160105-0000-001","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }
{"index":{"_id":"637471676729630133-06160106-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160106-0000-001","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }

In neither case is any delete called. If I remove the Where clause from this (default Accelerator code):
foreach (var channel in channels).Where(x => !usedChannelSystemIds.Contains(x.SystemId)))

Now, I see one delete per variant (+1 for base product), as expected. And switching between hidden / shown works.

POST http://localhost:9200/_bulk {"delete":{"_id":"637471676729630133-0616010cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"delete":{"_id":"637471676729630133-06160105-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"delete":{"_id":"637471676729630133-06160106-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"delete":{"_id":"637471676729630133-06160107-0000-001_Scbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"index":{"_id":"637471676729630133-06160105-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160105-0000-001","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }
{"index":{"_id":"637471676729630133-06160106-0000-001cbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160106-0000-001","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }
{"index":{"_id":"637471676729630133-06160107-0000-001_scbd3f0fc-5fdc-44a7-8106-1d4fe0eddf31","_index":"761-rev2productdocument.en-us"}}
{"articleNumber":"637471676729630133-06160107-0000-001_s","assortments":["66dab9b4-9f2a-4fb7-8994-fcec2d6072ed"],"baseProductSystemId":"c2a56b74-9934-4dc8-837f-51f6bcdc4deb", ... }

[Robin.w]

@NilsN

Below is the bulk sent when going from Yes to No on 1 variant. All 3 variants are expected to be indexed.

Yes, the accelerator code seam broken in this case. Also is the : IndexAction.Delete used at all in the accelerator? or is this used lower down in the Litiums DLL’s? It is set in the indexing events but dose not sesam to be used? (could be that i have old accelerator code)

[patric.forsgard]

The IndexAction.Delete should call the following method

But it is also existing logic that you cant have both delete and index in the queue at the same time, if that is needed you should use the IndexAction.DeleteAndIndex instead.

@Robin.w

Yes, the accelerator code seam broken in this case

It may be that when the code in the CreateModelsPerChannel method is added the corresponding code for filtering out the correct actions (index/delete) should be executed.

I haven’t test this but if you unpublish the variant on the channel, will the data still be in the index or will the data be removed in this case? If I look at the latest code I can see that we trigger IndexAction.DeleteAndIndex when we removing a variant but not when a variant is unpublished.

[Robin.w]

@patric.forsgard

I haven’t test this but if you unpublish the variant on the channel, will the data still be in the index or will the data be removed in this case? If I look at the latest code I can see that we trigger

It will be removed if we unpub, Can we change the action inside the CreateModelsPerChannel method? If we there find out that a varient should no longer be indexede can we set the IndexAction than? or do need to change the overall logic to always do DeleteAndIndex?

[patric.forsgard]

@Robin.w

It will be removed if we unpub, Can we change the action inside the CreateModelsPerChannel method? If we there find out that a varient should no longer be indexede can we set the IndexAction than? or do need to change the overall logic to always do DeleteAndIndex ?

Inside the CreateModelsPerChannel is to late to define the index action, this method is only executed when IndexAction.Index is there. The part that need to be adjusted is for the variants that not should be published on the channel to return the RemoveDocument item instead, then any existing items that may be in index will be removed.

Always ignoring the .Where(x => !usedChannelSystemIds.Contains(x.SystemId)) in the above code will make the items to be removed so that can be one option, but for index document that should be there it may be better to not send first a delete action and directly after the index action to minimize the work that the search engine need to to.

[Robin.w]

Oki, do i understand it correctly that the Enumerator can contain both RemoveDocument and ProductDocuments and the action determins what action to run/ignored?

The DeleteAndIndex will run booth? But using the Index
action willl ignore all RemoveDocument present in the list?

Same with DeleteAndIndex if I’d I return 0 RemoveDocuments it will not automatically remove all items before creating?

So if i removed the RemoveDocument part and run DeleteAndIndex nothing will be deleade?

Is this right?

[patric.forsgard]

IndexAction.Delete will call the method BuildRemoveIndexDocuments
IndexAction.Index will call the method BuildIndexDocuments

For IndexAction.DeleteAndIndex will first fall the method BuildRemoveIndexDocuments and then the BuildIndexDocuments and the purpose is to use this index action when the index need to remove the orphaned items before new should be indexed.

The enumerator can contain both the RemoveDocument and other documents, ex ProductDocument. The document will be processed in the order they are returned from the method except for the RemoveDocuemnt that always will be executed first to clean out information before other index operations.

[Robin.w]

haha I still don’t get it fully, if i do a “IndexAction.Index” action but add RemoveDocument to the list, will that document be removed or will it be ignored?

[patric.forsgard]

It will be removed.

[Robin.w]

@patric.forsgard @NilsN

I think that adding this line to the accelerator solves the problem.
Its not the easyest to read but it dose the jobb.

|| productDocuments.Where(y => y.ChannelSystemId == x.SystemId).ToList().Count < variants.Where(z => z.ChannelLinks.Any(zz => zz.ChannelSystemId == x.SystemId)).ToList().Count)

So we remove the products if the channel document count differes from the published variants.

// Remove all documents for all channel-combinations
			foreach (var channel in channels.Where(x => !usedChannelSystemIds.Contains(x.SystemId) || productDocuments.Where(y => y.ChannelSystemId == x.SystemId).ToList().Count < variants.Where(z => z.ChannelLinks.Any(zz => zz.ChannelSystemId == x.SystemId)).ToList().Count))
			{
				var cultureInfo = _languageService.Get(channel.ProductLanguageSystemId.GetValueOrDefault()).CultureInfo;
				yield return (cultureInfo, new RemoveDocument(new ProductDocument
				{
					ArticleNumber = baseProduct.Id,
					ChannelSystemId = channel.SystemId
				}));

				foreach (var variant in variants)
				{
					yield return (cultureInfo, new RemoveDocument(new ProductDocument
					{
						ArticleNumber = variant.Id,
						ChannelSystemId = channel.SystemId
					}));
				}
			}

Dose this seam like an ok soulution or will it cause problems? Adding remove actions for all variants and later dont the logic adding ProductDocuments, should remove and than add back right?

Edit: My code ended up like this:

			bool RemoveFromChannel(Channel channel)
			{
				var doseChannelHaveVarients = usedChannelSystemIds.Contains(channel.SystemId);
				var channelDocumentCount = productDocuments.Where(x => x.ChannelSystemId == channel.SystemId).ToList().Count();
				var activeVarientsOnChannel = variants.Where(x => x.ChannelLinks.Any(xx => xx.ChannelSystemId == channel.SystemId)).ToList().Count();
		
				return !doseChannelHaveVarients || (channelDocumentCount < activeVarientsOnChannel);
			}

[patric.forsgard]

Yes, if you want little less CPU cycles for each variant you should exit early, direct after

var doseChannelHaveVarient = ...

to not to the other calculations if channel not have any variants.