[Cod-bugs] Potential firewall block

[Antanas Vaitkus]

Dear Herman,

Thank you for reaching out. Recently our servers started being routinely
overwhelmed by search engine/AI platform crawler bots, therefore we
had to implement a slightly stricter automatic blocking policy. Your usage
does not seem to fall under this category therefore we will whitelist your
server IP. Could you please let us know if this fixed the issue on your end?

As for your suggestion, optimising your workflow to minimise the number
of queries is always welcome (as well as pausing between several queries).
However, in this case the CIF download itself is probably the more expensive
query (could please provide some specifics on how you download the files?).
You are welcome to continue downloading COD files using your current
approach,
but I would also recommend looking into downloading the entire COD dataset
and serving it from a local copy (see
https://wiki.crystallography.net/howtoobtaincod/).
Specifically, the Subversion or rsync approaches would be the easiest to
keep in
sync and up to date.

Sincerely,
Antanas


On Wed, 21 May 2025 at 12:28, Herman Bergwerf <post at hbergwerf.nl> wrote:

> Hi,
>
> I am experiencing some issues with using COD on molview.org; we are
> accessing the COD MySQL database and the crystallography.net cif files,
> but these are no longer working and we are wondering if the server we
> are hosting molview.org on is being blocked by the firewall.
>
> Our server IP is: 68.66.248.52
>
> molview.org is on shared hosting without SSH access, so sadly I have
> very limited access for debugging this (but I can ask the support team
> if needed).
>
> I can imagine there were unreasonably many requests to the MySQL
> database coming from our server. I quite inefficiently programmed the
> CIF retrieval script to first check in the database if the COD record
> exists and has coordinates, and only after that do I attempt to
> download the CIF file. I can change this to just trying to download the
> CIF file, which should significantly reduce the load you are
> experiencing from molview.org
>
> Thanks,
> Herman Bergwerf
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
> _______________________________________________
> Cod-bugs mailing list
> Cod-bugs at lists.crystallography.net
> http://lists.crystallography.net/cgi-bin/mailman/listinfo/cod-bugs
>


-- 
Antanas Vaitkus,
Vilnius University,
Life Sciences Center,
Institute of Biotechnology,
room C521, Saulėtekio al. 7,
LT-10257 Vilnius, Lithuania

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.crystallography.net/pipermail/cod-bugs/attachments/20250521/26887ddc/attachment.htm>