3

TM search does not catch sentences within long segments

Julia Rubio

February 07, 2020 10:40

Answered

Follow

Hi! 

 

Does anyone know with the TM search within the Editor or the Global search cannot find sentences within long segments?

Due to character restriction issues, we often need to merge segments, resulting in big chunks of text within a single segments.

Unfortunately, our experience shows that, even if we merge these segments into the general TM, then they cannot be searched for in the Editor translation finder or in the Global search. This results in our team thinking that the segments have been removed from the TM,then having to retranslate the sentence again from the scratch and compromising consistency.

However, when opening the TM from the Resources section we can verify that the segments have always been there.

 

Is anyone experience a similar issue? Could anyone from the Wordbee team show us how to fix this situation?

 

Thanks!

• Facebook
• Twitter
• LinkedIn

6 comments

• 2
  
  Gregory Pageau August 20, 2020 15:24

  Hi Julia,

  I do experience the same issue. 

  From a general point of view, the TM search in Wordbee is very weak because it doesn't always show or highlight in yellow the most relevant matches (while they do exist, being a word, a group of word or an entire sentence) and doesn't sort them either by order of relevance. So you end up not knowing what's really available in the TM and having to translate it all over again, underming consistency, as you rightly pointed it out.

  Looking at the date you posted your comment (more than 7 months ago!!!), I am surprised though that nobody from the Wordbee team took the time to reply to you!

  Cheers

• 2
  
  Mike Holland September 04, 2020 15:12

  Hi Julia and Gregory,

  Yes, it would be good if WB at least responded to this. 

  I've found that the presence of tags in a source segment also impedes WB's search ability. If you copy the source text from the MT field (which shows the text with the tags stripped out), and paste it into the search field, you can sometimes find a match that didn't show up with the tags included.

  Term search is weak as well (in Japanese to English direction).

  Best,

  Mike

   

• 0
  
  Mike Holland September 23, 2020 20:41

  I still encounter many instances of terms in our termbase not being shown in the translation finder.

  I wonder what causes the problem and what WB is going to do to fix it?

  Mike

   

• 0
  
  Harry van Horen July 14, 2023 10:56

  I understood in the past that when a term or even sentence is less than 60% of the complete segment in the tm the translation finder will not show it. No idea why they cannot resolve that.

  Harry

• 0
  
  Maik Mehlhose July 14, 2023 12:54

  Dear Julia, Gregory, Mike & Harry, 
  
  Harry, thanks for reactivating this post. Let me go through the various issues one by one and explain what solutions we've put in place and are currently planning to improve the editor's behavior.

  1. Rubio: Does anyone know why with the TM search within the Editor or the Global search I cannot find sentences within long segments?

  Harry is kind of right about this, but let me elaborate a bit. When you open the Translation Finder settings, you can set a threshold for displaying TM hits (see below). If you set the threshold to 80%, as in the example, the Translation Finder will only show hits that have at least 80% similarity to the source text. You can lower this threshold to 50%, which may help in some cases, but will generally result in more noise in the displayed results.

  

  For example, if you have long segments in your translation memory that consist of three sentences of approximately the same length, and you encounter a single sentence in a source segment, the similarity is technically only 33%, which is well below the 50% threshold you can set. That's why it is recommended to keep the segment size on sentence level whenever possible.

  I realize that this is not always possible. Currently, you can use the Concordences search in the translation finder to find your TM hits. Let's take a look at how the translation finder works:

  Whenever you enter a segment, Wordbee takes the entire source segment and searches the whole database of attached TMs and TBs for hits. As you can see in the example above, all hits that are not similar enough will be discarded. What you can do in the Translation Finder, however, is to start your own search. You can take a single sentence out of the segment and search for it instead of the whole segment. This can do one of two things. Either you find a 100% memory hit for that specific phrase and can reuse it, or you find that phrase within a long segment consisting of multiple phrases. In the latter case, you will find the translation in the Concordances tab. The same applies to Global Search. If you split your segments and search for the individual phrases, you will be able to find the relevant hits.

  

  I understand that this is far from ideal at the moment, but perhaps the search will help you find some of the relevant strings.

  2. Harry: No idea why they cannot resolve that

  Searching databases is quite resource intensive and requires specialized search platforms to facilitate the technical process. Some of our customers have multiple resources attached to projects with tens of thousands of term base entries and millions of strings in translation memories that need to be searched for every single string a customer enters. We simply have to set a limit somewhere to ensure that users get relevant results quickly.

  Also, just because one string is 30% similar to another doesn't mean that the match is useful. In the case where there is exactly the same phrase within a segment, it is useful, while the right mix of words like "is, you, the, of, and, at" and so on can also result in a scattered 30% match that isn't really relevant to the translator's work.

  This also relates to the point Gregory made: The hits appear to be ordered without relevance, when in fact the order for low similarities and phragments is based on the mathematically applied search algorithm with higher similarities at the top.

  I will check with our development team to see if we can do something here, like sending the original big string to a search and then also the individual sentences within a segment, to see if we can resolve this issue and separate the good 30% hits from the useless ones.

  3. Mike: I've found that the presence of tags in a source segment also impedes WB's search ability.

  I think this is simply a matter of your resource settings. In the search settings of a resource, you can change the weight given to certain properties (including tags) when calculating similarity. You should get rid of your problem by setting the "Penalties for formatting differences (inline tags)" option to a lower setting or none.

  

  4. Mike: I still encounter many instances of terms in our termbase not being shown in the translation finder. I wonder what causes the problem and what WB is going to do to fix it?

  The more segments in the attached TMs and terms in the attached termbases, the more hits will be generated. If there's a segment with a lot of similar hits and terms, our search platform has to reduce the amount of information in order to deliver relevant results quickly. This problem only occurs with extremely large amounts of strings and term bases, such as those that accumulate over a decade or more of using TMs and term bases within a large organization.

  Since you asked this question in 2020, we have significantly improved our search algorithm to better handle these resource-intensive searches. Most of our customers who had these issues 3 years ago don't have them today.

  Wordbee users can also help avoid these issues by curating their translation memories and databases. Redundant or obsolete strings or terms should be deleted or moved to legacy memories so that we can technically limit the search to relevant entries and improve search results.

  To further mitigate this problem, we have conducted tests and outlined an approach to upgrade our search platform, which is responsible for the terminology search. The upgrade will make the technical search process more efficient and make sure that relevant results are displayed. The upgrade will need to be rolled out individually for each existing Wordbee platform to avoid long downtime and will take several months to complete. We're currently discussing the appropriate timelines.

  I hope these explanations help a little bit. If you have any more questions or insights, please let me know. The Wordbee forum has been neglected for a few years and I only picked it up in December 2020. If there's any other post from before this time you seem deserves an answer, please feel free to reactivate it again.

  Cheers,
  Maik
  
  Maik Mehlhose
  Head of Product

• 0
  
  Maik Mehlhose July 28, 2023 07:09

  Hello everyone,
  
  I just checked with our development team regarding this part in my iriginal answer:

  "I will check with our development team to see if we can do something here, like sending the original big string to a search and then also the individual sentences within a segment, to see if we can resolve this issue and separate the good 30% hits from the useless ones."

  Apparently we will have to deploy the enhancements for the terminology search first that will then allow us to move forward with these search enhancements. I will update you here once we make progress on these projects.

  Cheers,
  Maik
  
  Maik Mehlhose
  Head of Product