<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Dear William,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">thank you very much for the answer and
for the COD processing logs! They are indeed very useful for us.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Below, I give some comments on the
kinds of issues your software has found. Some are hard to fix and
reflect the real situation in the crystal, or the choice and
opinion of the structure author. Some, however, are serious
problems which we missed and which we will have to investigate. <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Details below:<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2022-11-17 18:13, William Lenthe
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">Please find a list of the errors generated by my parsing of the database. I've manually removed a few types of errors that are due to limitations in my code. For example I don't handle modulated structures (I count 285 in the database) </pre>
</blockquote>
I found 287 in the recent revision of the COD, which is probably the
same as yours with just some new structures. Confirmed.<br>
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">or support cifs without a space group (15), lattice parameters (10), and at least 1 atom (there are so many of these I didn't count). I've also removed 286 instances of an error message like:
1501515.cif: _atom_type_symbol 'Ti4+' not found in _atom_site_label loop</pre>
</blockquote>
<p>I get similar counts; confirmed. The _atom_type_symbol needs to
be fixed systematically, somehow, but there are 115772 such
messages, so hard to go through manually.</p>
<p>The '_atom_site_type_symbol' value Ti5+ seems to be a mismatch
with the '_atom_type_symbol' value 'Ti4+'; this is probably a
mistake but we need to check the original paper before correcting.
</p>
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">Since it looks like the database you send already catches that type. I left in the case sensitive versions since they should be easy fixes. Finally also have a list of ~35k warnings that are almost all one of these types:
1006173.cif: clamped site O2 occupancy from 1.005(5) to 1.</pre>
</blockquote>
Here, the occupancies seem to be refined. The occupancies are both
within error margins of 1.0. The occupancies of oxygens O1 and O2
add up to 2.0, so clamping one but not adjusting the other will give
slightly incorrect total oxygen count in the structure. Whether this
is an issue will depend on your application, of course... We'll
leave these values as they are in the COD since this is how authors
reported the structure; of they refined the occupancies we need to
leave indication that this was done so.<br>
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">
1000495.cif: merging equivalent positions to 0.365000 0.365000 0.375000 with total occupancy 0.25 from sites labeled {Cs1, Cs2}.</pre>
</blockquote>
Specifying multiple atoms at exactly the same site seems to be an
accepted way to represent occupational disorder. I would say this is
a feature, not a bug ;)
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">1008070.cif: space group is triclinic but lattice constants are orthorhombic.</pre>
</blockquote>
<p>Monoclinic space groups can gave (nearly) any angles, including
90.0 degree angles. It would be of course strange to see
monoclinic cell with all right angles just by accident, but in
this case the abstract of the paper [2] says:</p>
<p>"For <em>x</em> > 0.9, these compounds have an orthorhombic
symmetry (<em>O</em>) if the cations are disordered, while the
symmetry lowers to monoclinic (<em>Mβ</em>) if the cations are
ordered"</p>
<p>Thus, there is either a re-intepretation of orthorombic data as
monoclinic, or a transition between ordered and disordered phase
here, which do not change the cell angles. Thus I would say the
angles are legit.<br>
</p>
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">
1100066.cif: corrected trigonal/hexagonal unequal a/b from 9.048(1)/9.047673 to 9.047836.</pre>
</blockquote>
The cell dimensions are within error margin of each other, so we
probably leave them as the authors have reported them. Of course
your software is absolutely correct to merge the values if that is
needed for your application. <br>
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">
1503454.cif: corrected monoclinic b alpha from 89.990(6) to 90. corrected monoclinic b gamma from 89.995(6) to 90.</pre>
</blockquote>
Again, angles are within the specified error margins, and probably
were refined (not fixed), so we leave them as they are in the COD.<br>
<blockquote type="cite"
cite="mid:SN1PR07MB3902F38FE414724F8B49081288069@SN1PR07MB3902.namprd07.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">
The file is a ~11mb so I'll send it using a file service (hightail) instead of as an attachment</pre>
</blockquote>
<p>Thanks, I have stored it in our private repository, and we will
consult the file when we have auotmatic procedure to fix some of
the issues, or when we look for the issues that can be fixed
manually... <br>
</p>
<p>This is the catalogue of messages that I have extracted:</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ awk -F: '{print substr($2,2,12)}' cod_wrn.txt |
sort | uniq -c | sort -nr -k1,1 | cat -n<br>
1 33414 merging equi<br>
2 967 clamped site<br>
3 277 _atom_site_a<br>
4 224 space group <br>
5 219 no space gro<br>
6 86 corrected tr<br>
7 84 corrected mo<br>
8 74 corrected te<br>
9 51 corrected or<br>
10 22 corrected he<br>
11 21 corrected cu<br>
12 1 corrected rh<br>
</font></blockquote>
Below, I analyse all unique messages from 'cod_err_flt.txt':</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ awk -F: '{print substr($2,2,20)}'
cod_err_flt.txt | sort | uniq -c | sort -nr -k1,1 <br>
124 _atom_site_aniso_lab<br>
51 failed to unambiguou<br>
30 cif block has confli<br>
17 failed to parse spac<br>
5 cif block contains l<br>
3 cif block tag '_chem<br>
1 monoclinic groups mu<br>
1 line 86 isn't commen<br>
1 line 55212 isn't com<br>
1 line 490 isn't comme<br>
1 line 132 isn't comme<br>
1 line 128 isn't comme<br>
1 cif block loop has m<br>
1 cif block has loop r</font><br>
</blockquote>
</p>
<p>The '_atom_site_aniso_lab' message is genuine validation warning;
we'll triage and probably fix in due time, if it is possible to
fix at all.</p>
<p>The 'failed to unambiguously determine space group setting'
message is correct, there seem to be no symmetry operators nor
Hall symbol in these files. International Tables would imply a
default setting, but this might be dangerous to assume. Probably
we can not fix this unless authors confirm the setting hey used,
or the setting is recorded in the paper.</p>
<p>The 'failed to parse space group from string' message is correct;
these are either incorrectly recorded space groups from the
original CIFs (which we can not fix, probably), or modulated
structures which have superspacegoup name instead of the space
group name; these we can probably fix some day when we have
superspacegroup reduction to spacegroup code integrated into the
COD pipeline.<br>
</p>
<p>The following finding is for me quite worrying:<br>
</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ grep 'cif block has conflicted hall symbol'
cod_err_flt.txt | head -5<br>
1010928.cif: cif block has conflicted hall symbol (-P 3* 2n)
and space group operators (recovered p_3*_2_-1n)<br>
1010956.cif: cif block has conflicted hall symbol (-P 2n 2a)
and space group operators (recovered p_2bc_2ac_-1ac)<br>
1010962.cif: cif block has conflicted hall symbol (-P 3* 2n)
and space group operators (recovered p_3*_2_-1n)<br>
1011149.cif: cif block has conflicted hall symbol (-P 2n 2a)
and space group operators (recovered p_2bc_2a_-1a)<br>
2002944.cif: cif block has conflicted hall symbol (-P 4 2ab)
and space group operators (recovered p_4_2ab)</font><br>
</blockquote>
</p>
<p>Indeed, the Hall symbols and the symmetry operators in the
structures do not match (30 cases). We'll have to look at the
original publications to find out why this is so. We'll add the
code to check symop-Hall symbol correspondence to our COD check
routines. Thanks for pointing this out!</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ grep ': line' cod_err_flt.txt<br>
4029286.cif: line 490 isn't comment or part of loop row of cif
but doesn't have _<br>
7223602.cif: line 86 isn't comment or part of loop row of cif
but doesn't have _<br>
7228312.cif: line 128 isn't comment or part of loop row of cif
but doesn't have _<br>
7238658.cif: line 132 isn't comment or part of loop row of cif
but doesn't have _<br>
7705257.cif: line 55212 isn't comment or part of loop row of
cif but doesn't have _</font><br>
</blockquote>
</p>
<p>It is not quite clear what this error message is saying but yes,
all these cases are not comments and not parts of loops; they are
parts of multi-line text fields delimited by ';' tokens and can
contain arbitrary text (well, nearly arbitrary). Could it be that
your CIF parser misses the beginning of a text field?</p>
<p>Of these files, the 7705257.cif contains a garbled HKL Fobs
reflection list, the rest seem OK.</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ grep 'cif block contains l' cod_err_flt.txt<br>
4002451.cif: cif block contains loop_ not followed by any tags
at line 59<br>
4130765.cif: cif block contains loop_ not followed by any tags
at line 943<br>
7035327.cif: cif block contains loop_ not followed by any tags
at line 6035<br>
7035331.cif: cif block contains loop_ not followed by any tags
at line 6694<br>
7035332.cif: cif block contains loop_ not followed by any tags
at line 6802</font><br>
</blockquote>
</p>
<p>These files (as all COD files) are syntactically OK and tags do
follow the 'loop_' token. Cold it be that you parser fails to
discard spaces at the beginning of the line before the tags?</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ grep "cif block tag '_chem" cod_err_flt.txt<br>
4034776.cif: cif block tag '_chemical_name_systematic'
followed by new line and quoted string at 36 but quoted string
doesn't close or fill entire line<br>
7201872.cif: cif block tag '_chemical_name_common' followed by
new line and quoted string at 41 but quoted string doesn't
close or fill entire line<br>
7233594.cif: cif block tag '_chemical_name_systematic'
followed by new line and quoted string at 37 but quoted string
doesn't close or fill entire line</font><br>
</blockquote>
</p>
<p>The situation is funny with these files. Syntactically, they are
correct – the CIF syntax [1] permits any <NonBlankChar> as a
trailing character of a <UnquotedString>, including a single
quote ("'")! So your parser misleads us: there is no quoted string
in these cases, but an <i>unquoted</i> string instead that is
terminated with a quote (which is a part of the value).<br>
</p>
<p>However, chemical names in all these files seem to have a
superfluous trailing quote that should not be included in the
name. We should curate these entries according to the chemical
names given in the respective papers. Our CIF checker
(cif_cod_check) could probably issue a warning when such strange
chemical names are encountered...</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ grep 'cif block loop has m' cod_err_flt.txt<br>
4301644.cif: cif block loop has multi line delimeter token mid
line at line 134</font><br>
</blockquote>
</p>
<p>Again, this is OK syntactically but does not represent the
intended data. CIF syntax is weird... Another check needed for
'cif_cod_check'?</p>
<p>I have fixed the entry in the COD :).<br>
</p>
<p>
<blockquote type="cite">saulius@tasmanijos-velnias 2022-11-17/ $
grep 'cif block has loop r' cod_err_flt.txt<br>
4342694.cif: cif block has loop row with 4 columns at line 153
but loop has 2 column headers<br>
</blockquote>
</p>
<p>This is a false alarm; the loop is perfectly OK:</p>
<p>
<blockquote type="cite"><font face="monospace">saulius@tasmanijos-velnias
2022-11-17/ $ sed -n '152,153p' $(codid2file 4342694)<br>
_space_group_symop_operation_xyz<br>
1 x,y,z 2 -x,-y,-z</font><br>
</blockquote>
</p>
<p>Typing two data packets ('loop rows') in one physical line is
perfectly OK in CIF.</p>
<p>Regards,<br>
Saulius<br>
</p>
<p>Refs.:</p>
<p>[1] IUCr. CIF v1.1 File Syntax. URL:
<a class="moz-txt-link-freetext" href="https://www.iucr.org/resources/cif/spec/version1.1/cifsyntax#gram">https://www.iucr.org/resources/cif/spec/version1.1/cifsyntax#gram</a>
[accessed 2022-11-18T18:23+02:00].</p>
<p>[2] <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-size: medium; font-style: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; display: inline !important; float: none;">Muller,
J.; Joubert, J. C. & Marezio, M.</span><br
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-style: normal; font-variant-caps:
normal; font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-size: medium; font-style: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; display: inline !important; float: none;">Etude
des phases du système FeVO4–VO2, obtenues par synthèse
hydrothermale à 70 kbar et 1000textdegreeC<span
class="Apple-converted-space"> </span></span><br
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-style: normal; font-variant-caps:
normal; font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;">
<em style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-variant-caps: normal; font-weight:
normal; letter-spacing: normal; orphans: auto; text-align:
start; text-indent: 0px; text-transform: none; white-space:
normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;">Journal
of Solid State Chemistry,<span class="Apple-converted-space"> </span></em><span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-size: medium; font-style: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; display: inline !important; float: none;"></span><em
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-variant-caps: normal; font-weight:
normal; letter-spacing: normal; orphans: auto; text-align:
start; text-indent: 0px; text-transform: none; white-space:
normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;">Elsevier
BV,<span class="Apple-converted-space"> </span></em><span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-size: medium; font-style: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; display: inline !important; float: none;"></span><b
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-style: normal; font-variant-caps:
normal; letter-spacing: normal; orphans: auto; text-align:
start; text-indent: 0px; text-transform: none; white-space:
normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;">1976</b><i
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-variant-caps: normal; font-weight:
normal; letter-spacing: normal; orphans: auto; text-align:
start; text-indent: 0px; text-transform: none; white-space:
normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;">, 18</i><span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);
font-family: sans-serif; font-size: medium; font-style: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; display: inline !important; float: none;">,
357-362, DOI: <a class="moz-txt-link-freetext" href="https://doi.org/10.1016/0022-4596(76)90118-3">https://doi.org/10.1016/0022-4596(76)90118-3</a><span
class="Apple-converted-space"> <br>
</span></span></p>
<p>Sincerely yours,<br>
Saulius<br>
</p>
<pre class="moz-signature" cols="72">--
Dr. Saulius Gražulis
Vilnius University, Life Science Center, Institute of Biotechnology
Saulėtekio al. 7, LT-10257 Vilnius, Lietuva (Lithuania)
phone (office): (+370-5)-2234353, mobile: (+370-684)-49802, (+370-614)-36366
</pre>
</body>
</html>