Dear Stefaan, dear colleagues,
On 2013-09-06 19:02, Stefaan Cottenier wrote:
(as it seems there have been technical problems about TCOD, I'm not sure whether this 3-months-ago mail actually reached all or you or not. I just resend it.)
Many thanks for reposting your e-mail. The TCOD mailing list, indeed, had a configuration error, but now I hope I have fixed it and our discussions can continue unhindered.
-------- Original Message -------- Subject: some discussion topics related to TCOD Date: Fri, 24 May 2013 14:34:08 +0200 From: Stefaan Cottenier Stefaan.Cottenier@UGent.be To: tcod@lists.crystallography.net
Dear colleagues,
Today, we held a group discussion at CMM (http://molmod.ugent.be) about TCOD from the perspective of users and/or structure donators. That did not lead to clear conclusions, but rather to a series of thoughts that can be a starting point for further discussions or actions. I'll list those here (I guess that is what this mailing list is meant for):
Thanks a lot, your input is very important, and we definitely need to take into account everyone's needs and opinions to make TCOD useful.
- You are probably aware of other database initiatives for computed
crystal structures. Is there a vision on whether TCOD wants to 'compete' with those, or whether TCOD tries to fill a niche that is not served by other databases?
For instance, consider https://www.materialsproject.org/. This is a database that aims doing a VASP geometry optimization for every structure in ICSD, starting from the ICSD cif (quite ironically, you can read their starting geometry (=ICSD info) for free, without having an ICSD yourself...). They use their own dedicated supercomputer infrastructure to run all this, and upload only results achieved by themselves. No input by others, this in order to keep control over quality and consistency. It has over 30.000 crystals by now, and apart from the structure info also computed properties are being added.
Sure enough, there are several approaches besides TCOD already that implement collections in theoretical structures in one way or another. I have read a bit about the https://www.materialsproject.org/ and I am still familiarising myself with others you have mentioned in your 2013-08-20 01:44 e-mail (I would like to encourage you to repost it to the TCOD list if you do not object making it open).
The main difference of TCOD is that, unlike the Materialsproject group who as you say "pload only results achieved by themselves", TCOD is (*already* :) open to wide range of contributors, including the Materialsproject if they wish or agree to share their data on the Open Data basis. This may be both TCOD's weakness and strength. As you say, "No input by others, this in order to keep control over quality and consistency" -- true. But on the other hand, the scientific community is broader and is getting broader, there are more programs than VASP (even if VASP would be considered the best), and it would be interesting to compare various calculations, and calculations vs. experimental results -- a COD/TCOD/PCOD bundle would make this easy to accomplish if done properly.
I quote here a paragraph from a project proposal which we submitted earlier this year, and which contains info/references about other databases:
"As is often the case for revolutions, this idea has been realized simultaneously and more or less independently at several places, emphasizing different aspects. The Materials Project [Jai11,MP] is an initiative at MIT, where basic properties of all crystalline solids documented in the (experimental) Inorganic Crystal Structure Database (ICSD) [ICSD] have been computed by DFT. The computed formation energies are used to construct secondary data bases of binary and ternary phase diagrams [Ong08, Jai11b]. Similar initiatives, each with their own focus and in different stages of growth are AFLOWLIB [Set10,Cur12] (Duke University), OQMD [Wol12] (Northwestern University) and CompES [CES] (NIMS, Japan). Other database initiatives that emphasize collaborative data sharing (see also Sec. 2.3, WP3) are ESTEST [Yua10,Yua12] at UC Davis (US), the Computational Materials Repository (CMR) [Lan12] at DTU (Denmark) and the AIDA environment [Koz13]."
[Jai11] A. Jain et al. Computational Materials Science 50 (2011) 2295 [Jai11b] A. Jain et al., Physical Review B 84 (2011) 045115. [Ong08] S. P. Ong, L. Wang, B. Kang, G. Ceder, Chemistry of Materials 20 (2008) 1798 [Set10] W. Setyawan, S. Curtarolo, Computational Materials Science 49 (2010) 299 [Cur12] S. Curtarolo et al., Computational Materials Science 58 (2012) 227 [Wol12] Open Quantum Mechanical Database, http://wolverton.northwestern.edu/oqmd (no public access yet) [CES] http://caldb.nims.go.jp/index_en.html (Y. Chen, A. Nogami, H. Ohtani, N. Tatara) [Yua10] G. Yuan, F. Gygi, Computational Science & Discovery 3 (2010) 015004 [Yua12] G. Yuan, F. Gygi, Computer Physics Communications 183 (2012) 1744 [Lan12] D. Landis et al., Computing in Science and Engineering, nov/dec 2012, p. 51 (http://dx.doi.org/10.1109/MCSE.2012.16) [Koz13] B. Kozinsky, N. Marzari, N. Bonini, J. Garg, G. Pizzi, A. Cepellotti, M. Fornari, contributions at the MRS Fall meeting (Boston, Nov. 2012) and the DPG Spring meeting (Regensburg, March 2013).
You are right, the idea of having Open theoretical structure database is floating in the air. But is there a database that is:
a) open (as in Open Data), permitting unrestricted sharing, reuse and republication of data if cited properly; b) accepting contributions from all relevant sources; c) up and running right now?
If yes, then TCOD is probably not needed. Indeed, if Computational Materials Repository accepts the computed depositions, then its probably the one that does the job. If, however, it does not -- we give a try, why not? The need is there, and the TCOD was basically a reaction to the request from community to deposit computed crystal structures which COD currently either rejects or checks poorly.
(2) From our perspective, we don't understand the need to separate experimental and computational databases. As long as every structure is properly tagged as 'experimental' or 'computed', we see no reason to separate them. It will only lead to the burden of having to run queries twice. In any case, a unified search web page that searches in both data bases simultaneously seems useful to us (that is, de facto, treating (T)COD as one database).
These concerns will be addressed. The division COD and TCOD is pure technical; searches will be possible in the uniform unions of COD ad TCOD and PCOD.
The reason for separation are the different data requirements for experimental and computed structures -- COD needs criteria that show how good the model matches the observed data (R-factors, CC's, finally, the Fobs data itself); TCOD needs computational parameters that indicate convergence of the process, etc.
(3) Quality control. If a deposited computed structure has been published, the reference to the publication serves as quality control.
Absolutely. The COD/TCOD framework does store full provenance information in the records, and we treat published structures as peer-reviewed.
Still, "peer-reviewed" dos not always mean "correct" or "accurate"; especially when it comes to data...
That's probably similar for deposited experimental structures, isn't it?
Yes it is.
If a computed structure has not (yet) been published, how to assess its quality? Well, let us turn the question around: if a not yet published experimental structure is deposited, how do you judge its quality...?
For experimental structure, what counts is its correspondence to the observed data (Iobs/Fobs) and to our background knowledge (reasonably chemistry). The correspondence to data can be expressed in numeric terms in many cases -- Rcryst, CC, GoF are parameters that are routinely cited in experimental CIFs to show the size of discrepancies between the model and the experiment. No single parameter is perfect, for sure, and most have been criticized, but they do give a rough (and, most importantly, machine-processable) picture of the model-observation fit.
As for "reasonable chemistry", parameters are less formalised, but an experienced chemist can detect a lot of suspicious cases by looking at the structure, and we have now compiled comprehensive statistics on COD (the master thesis work of Andrius Merkys) so that we can now detect "low probability" structures automatically.
It makes sense to do it in the same way: as long as basic information is provided on how the calculation has been done, later users should make their own judgement.
Absolutely. But the numeric quality criteria (QC) are needed so that we can monitor the incoming structures automatically -- no-one will be willing or able to sift manually through the million+ structures, right?
(4) A quality control measure that could make sense for both experimental and computed structures, is to allow people to add remarks that appear on the web page of that structure ('considering the small basis set that has been used, I do not trust this result', or 'this result has been contradicted by <ref>').
You hit the nerve! That's what we are doing right now for COD, and TCOD will immediately benefit from the system as well.
I would like to point out however that community consensus does not necessary mean a correct judgement, just a widely accepted one. We have all knew for a couple of hundred of years that crystals do not have fivefold symmetry axes -- and yet Shechtman discovered quasicrystals, now well confirmed (right?).
Another problem with community review is that we simply do not have time to look through each and every file carefully enough. So computers should help us; my strategy would be to flag structures automatically as being "usual" or "unusual", and then ask people whether they find the provided evidence "convincing" or "unconvicing", ideally in the way you have proposed, 'this result has been contradicted by <ref>', or otherwise with a sound argumentation.
The "unusual" and "unconvincing" structures will probably need to be reinvestigated; the "unusual" but "convincing" ones might be a real gems!
I hope this strategy would be applicable both for COD and TCOD.
Such information by the community can help people who are not experts in either computing or measuring crystal structures to decide to which extent they can trust a particular result.
Sure. Lets present it on the *COD search results? Group structures by "reliability index"?
(Another variant is a feature on the web page of each structure to flag it if you have doubts, such that experts can look at it).
Will be done automatically in a near future.
(5) COD has cifs as entries only. If you want to go as far as including input info to reproduce each calculation, it might be hard to stick cifs-only for TCOD. It will then be necessary to link additional files to the cif of each entry.
Adding additional files is possible; the upload site needs to be fixed to accept ans store arbitrary archives, but this is easy to do.
For TCOD, I envisage possibility to store files in arbitrary formats. O good strategy, IMHO, is outlined Tim Berners Lee’s Open data definition (I learnd about it from the Peter Murray-Rust's talk, http://www.iucr.org/__data/assets/pdf_file/0018/80280/murrayrust3.pdf). We can start accepting additional files that outline computation process, and later formalise data presentations and formats.
Of course, the better defined the formats are the more useful these auxiliary files will be. It would be nice if the community agreed on comprehensive and open computational data formats, and deposit also descriptions of their computational work flow, not just the final results. If this goal is attainable, remains to be seen...
(6) We are split over whether it makes sense to require that each deposited calculation is exactly reproducible.
So what if we say that at the moment this is "nice to have", but not a "must have" option?
Some basic information should be given, of course.
Agreed. And it is vital that the community agrees what information is necessary, and we formalise it in some ontology. CIF dictionaries which you have started are a good possibility for this; other ways are also possible.
This will be mainly information about the method that does not depend on the specific implementation (code). Furthermore, the main technical input parameters that are specific for the code used, should be given as well (if you want to do that in a structured way, you will immediately hit the problem that the required set of numbers will be a different kind of set for every other code...).
IUCr does similar thing for the crystal structure refinement data: they just include the input script of the refinement program verbatim, in the data item "_iucr_refine_instructions_details"; vis:
http://www.crystallography.net/cod/2234930.cif
Maybe such approach would be viable for DFT computations? Say in _dft_computations_details data item?
Third in line should then be a free field for 'any other special settings' that could apply. With this kind of info, people can reproduce to a large extent (95%) the same calculation. If you want to have it reproduced exactly, then much more input (files) will be required. Which might not be worth the effort.
Could be. Actually, storing the just the information you have outlined, and not the complete input files, would make our database maintainer's life easier as well.
Would a program, something like Torbjörn's 'cif2cell', be able to reconstruct at least approximate input scripts to repeat the computation? If yes, this would be probably what we really need.
(7) One of our enthusiastic PhD students with many years of experience in web design and databases immediately had a mental map about how to implement an automated version of (6) -- if the TCOD-team is willing to take him aboard to code that, he'll probably agree ;-).
Sure! Please invite him/her! Cordially welcome!
(8) In any case, it should be clear for the one who provides the structure, what exactly is requested.
A part of this would be CIF dictionaries and other ontologies, supplemented by deposition check software, supplemented by *good* documentation (hopefully ;).
A web form where the information under (6) can be filled out (or the automated tool of (7) that extracts all this info from the output of each mainstream code) will be required in order to have consistent data.
Good idea!
(9) Which kind of computed structures will TCOD accept? Ground state structures, obviously?
Sure!
Metastable structures probably as well?
Why not, if marked or otherwise detectable as such.
Also if they are not dynamically stable (soft phonon mode)? (that is not routinely examined)
Sure, very interesting to see such beasts. Again, marked properly.
And what about transition state structures? (never observable in experiment, yet very useful to know -- and hard to find -- in order to understand reactions)
Absolutely, this is what many chemists and biologists (including structural biologists) are interested in. I guess most structural enzymologists would love to see what a transition state of the substrate-to-product reaction of their beloved enzyme is, wouldn't they?
If the latter are allowed, then they should be tagged as such.
For sure. Actually, your list is a good addition for the dictionary -- each of your suggestions might become an enumeration value in a data item describing the struct
Thanks for your input!
Will put together your and others' ideas as suggestions for action soon on this mailing list.
Regards, Saulius