[Cod-bugs] COD conversion with HighScore

Thomas Dortmann thomas at tdsonline.nl
Tue Jan 14 11:33:52 EET 2020

Hi Saulius,

thanks a lot for your answer!
The COD conversion runs nearly 100% now - for the water (oxygen) molecules
addresses by us. There are many more problematic atom_names, as shown in
your lists.

Our heuristics is slightly different from yours and does interpret many of
the problematic names too, but if this interpretation is right or wrong
nobody knows. Next to your lists provided in the email we also experience
many atom_names like ?, n or simply a dot or an empty space, which are not
interpreted at all by us.

I fully agree with your plans a) and b)! In the end no heuristics is error
proof. It will be a big step forwards and a great improvement on the COD
quality, when as many CIF's as possible carry atom_symbols next to the

We will fix the "OWat(n)" atom_name and hope to have our conversion ready
by the beginning of next week.

best regards,
Thomas Dortmann

On Mon, Jan 13, 2020 at 8:21 PM Saulius Gražulis <grazulis at ibt.lt> wrote:

> Dear Thomas,
> On 2020-01-13 12:43, Thomas Dortmann wrote:
> > a) concerning atom-labels:
> >
> > we fixed the "Wat" atom-label in our conversion and now the number of
> pattern containing the (wrong!) element Astatine is down from 1180 to  8!
> > These eight remaining patterns are all minerals, where water is coded as
> "OWat(n)" instead of "Wat";
> > these are the corresponding CIF's:
> > 9006364 - 9006368
> > 9014246
> > 9014312
> > 9016377
> This is good news. I'm very happy that your conversion software runs
> nearly 100%!
> On my side, I went through the "uncanonical" atom names in the COD
> (
> http://saulius-grazulis.lt/~saulius/.d981490889b10e82e8f6943bbfd569aaebf1c8c3/
> ).
> In the file "estimated-atom-types.lst", the first column is the
> estimated atom type, the second is the atom name in the corresponding
> CIF, and the third is the number of occurrences of this atom name.
> The "DOUBLE_CHECK.lst" contains a manually compiled list of atom types
> that are most probably wrong after automatic detection and will need to
> be inspected by a human.
> The policy I would adopt is the following:
> a/ If a CIF already contains _atom_site_type_symbol, we do nothing.
> Reason: the _atom_site_type_symbol is either added manually by COD
> curators (in this case we do not want to undo our manual work), or it is
> provided by CIF authors. Among the atom type symbols, most common
> irregularity is the symbols in all lowercase, or the symbols in all
> uppercase. These can be dealt by regularising case an looking up in a
> table; e.g. we do:
> ucfirst(lc($atom_type_symbol)),
> where lc($string) returns all-lowercase version of the argument string,
> and ucfirst($string) returns the string with the first letter
> uppercases, yielding "Ca" from both "ca" and "CA", which is mostly
> correct. From 14742 atoms in the COD that have _atom_site_type_symbol
> values, only 28 could not be interpreted in this way – a negligible
> amount, and non-correctable even manually.
> Since, as I understand, you software already incorporates this
> heuristics, atoms with _atom_site_type_symbol will not be a problem,
> will it?
> b/ If the atom does *not* have the _atom_site_type_symbol, we will guess
> its type from the atom label. If the leading non-digit characters of the
> atom label yield a valid periodic system element name, we do nothing.
> If the leading non-digit characters of the atom site label do *not*
> yield a recognisable atom name, we apply heuristics as noted in the
> estimated-atom-types.lst.log in the Web page cited above; in Perl:
> $n1 = ucfirst(substr($atom_site_label,0,1));
> $n2 = ucfirst(lc(substr($atom_site_label,0,2)));
> if( $atom_site_label =~ /^Wat[A-Za-z0-9\(\)]*$/ ) {
>     $atom_site_type_symbol = "O"
> } elsif( exists $COD::AtomProperties::atoms{$n2} ) {
>     $atom_site_type_symbol = $n2
> } elsif( exists $COD::AtomProperties::atoms{$n1} ) {
>     $atom_site_type_symbol = $n1
> } else {
>     $atom_site_type_symbol = "?";
>     print STDERR "$0: WARNING, atom type for atom \"$F[1]\" is not
> recognised\n"
> }
> We then compute the summary formula with the new atom types, and compare
> it with the formula provided by the authors. If the summary formulae
> match, we add the _atom_site_type_symbol to the CIF. If not, we report
> an error.
> After this, we double-check the atom types mentioned in DOUBLE_CHECK.lst.
> The new modified CIFs will have recognisable (standard) element names in
> _atom_site_type_symbol, and will have correct chemical formula
> computable from atom records (correct means the same as provided by the
> author). The results will be like to those in estimated-atom-types.lst.
> The new CIFs may only break the heuristics in you program if:
> 1/ we guess the atom types wrongly,
> 2/ the authors provided an incorrect summary chemical formula
> 3/ the two incorrect formulas match by pure accident,
> 4/ your heuristics gets atoms types correctly.
> or
> 1'/ we make two mistakes that compensate each other exactly (e.g. Ca->C
> on one site, and C->Ca on another site) and still get the correct
> formula with incorrect atom site assignments.
> I regard coincidence of these events highly unlikely. Also, when
> detected, the _atom_site_type_symbol values can be curated manually and
> will *not* be overridden again by automatic software.
> If you find such COD curation policy acceptable, we proceed with its
> implementation at some time in the future, and add it to our automatic
> pipelines (but without the manual check stage for every incoming file...).
> I CC this e-ail to the COD AB for discussion and eventual policy approval.
> Regards,
> Saulius
> --
> Dr. Saulius Gražulis
> Vilnius University Institute of Biotechnology, Saulėtekio al. 7
> LT-10257 Vilnius, Lietuva (Lithuania)
> fax: (+370-5)-2234367 / phone (office): (+370-5)-2234353
> mobile: (+370-684)-49802, (+370-614)-36366

This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.crystallography.net/pipermail/cod-bugs/attachments/20200114/c941614d/attachment.htm>

More information about the Cod-bugs mailing list