Check bibentry duplicate detection algorithm

[tlunet]

Didn't work properly on one entry, see ba20cb1.

Problems comes when there is already many articles with the same ID on the same year (very productive first author), and the letter increment for the ID do not check if the article is there already in the incremented IDs that already existing (turned out, it did for the example above).

Suggestion : use the google-style ID, that uses the {name}{year}{firstTitleWord} where firstTitleWord is the first word of the title containing more than 3(?) characters.
Would need a proper rework of the scripts, avoiding code duplication by centralizing code in a common module (like utils.py or something similar).

💡 Should be an opportunity to remove all accents from the bibtex IDs

[pancetta]

I think backward compatibility is important here. I get your point and yes, this is not great, but welcome to automation hell.

[tlunet]

Well, it's not hell once implementation is done properly 😉

But don't worry, I know how to solve this and will do that later ... with backward compatiblity

[pancetta]

Be my guest.

[tlunet]

But I'm not sure on "why" we need backward compatibility here ... is it just for people who copy paste the bib file and want to keep the same bibtex entries ? To be honest, I'm not sure this should be a reason 😅

[danielru]

Not sure how much backward compatibility you want to break? If this is just about removing the Bibtex IDs with accents, that should be fine. But I don't think changing every single identifier is a good idea.

[tlunet]

But I don't think changing every single identifier is a good idea.

Why ? Changing it to the google-based system would provide an easier way to generate unique Bibtex IDs and find duplicates. Furthermore, it won't change the ordering in the bib file, except eventually for the duplicates.

[danielru]

If all identifiers are changed, that will mean no manuscript written before or now with the current identifiers will compile without the user having to change every single citation in the paper. This is not impossible but I would only do that if there is a clear benefit.

[tlunet]

Why should a user change its own bib file ? The old refs can still be used, just new added reference will have the new identifier format, which is already done when other references are added (e.g from google scholar).
So old manuscript can still compile, as soon as they don't modify their original bib file (which I hope, is local and dedicated to the manuscript 😅 ... if not, that's a dangerous way of writing paper).

[danielru]

For old papers it may not be an issue but for papers still being written it is, because I update references by updating the pint.bib Bibtex file :D Again, I am not saying we cannot do it, just not clear what the benefit will be?

[tlunet]

Oh, I see 😄. Benefits of using a google scholar reference format {nameFirstAuthor}{year}{firstTitleWord} are then :

• rarely need to be incremented with a suffix, since it's rare that the same first author publish on the same year with the same first word on its title (larger than 3 characters).
• can be used to raise some warning when there is a first version of the paper one year (e.g with arxiv) then another version on the year after (e.g published in a journal), and eventually remove the old reference if it's indeed the same paper
• is easier to use in most latex editor, since the \cite command often have auto-completion and the first main word in the title allows you to find quickly which reference you need (e.g with gander[...] publications 😅 )

Practical example with the current references :

• WangEtAl2026 -> wang2026three
• WangEtAl2026b -> wang2026convergence
• WangEtAl2026c -> wang2026highly
• WangEtAl2026d -> wang2026parallel

For the last one, there was already a wang2024parallel corresponding to the arxiv version, so adding wang2026parallel could have raised a warning, which would have been used to decide which version to keep.

Note

I personally find the google scholar format quite convenient, especially when navigating through many references. And if they made it that way, there is probably a good reason 😇

[danielru]

Hm, okay, if this solves the issue with the a, b, c references, I am game. @pancetta You okay with that?

[pancetta]

What do you do in the (arguably) unlikely situation of an exact match, but different papers? I.e., if a (stupid) authors were to write two papers in the same year with the same first word in the title?

[tlunet]

This will happen only when a new DOI is included, so the bib bot can simply write a comment in the issue indicating the duplicate and all related details. Then we can decide :

• ignore if it's the same article but an older DOI (e.g arXiv) for the paper
• overwrite if it's the same article but a more recent DOI (published manuscript)
• increment it if that's two different articles : add a -2 as suffix (or more eventually)

With that, for the general procedure to add a new DOI, the bot generates the associated newRef and check if :

1. there is any existing reference starting with newRef in the database
2. check if there is any reference without year (e.g wang|parallel for wang2026parallel) that has a similar title (Levenshtein-based criterion)

If one of the point above occurs, then a warning comment will be issued and we will have to manually choose, as described above.