#!/usr/bin/perl
# fbdataminer.pl

=head1 FlyBaseDataMiner/101

FlyBase data reports with http://flybase.net/cgi-bin/fbidbatch.html

This is a simple perl example client that uses FlyBase.net batch ID query
function, which is *much* more efficient for the server to handle
than repeated calls to /cgi-bin/fbidq.html, the standard single report
url at FlyBase.

Please if you are writing data mining programs that call for flybase
reports by ID, use this /cgi-bin/fbidbatch.html with batches of IDs
instead of repeated single calls to the ID report url.

*** NOTICE TO DATAMINERS *****

FlyBase and other public bioinformatics servers have problems with 
not-nice data miners causing the server to go out
of service for all users, and must take steps to prevent this.
All bioinformatics students  need to learn the first
lesson in datamining on someone else's public web server is to be very
careful about over-use.  These are shared services that many thousands
of people use and rely on to be always working 24hr, 7days a week.

The average perl or java program can easily cause a server to die,
acting like a denial of service attacker, because they run as rapidly as
they can, without regard to shared usage needs. Often, as with
flybase.net, the web cgi-bin functions are designed for human use, and
unless they specially tuned for robots, these functions can be  costly in
compute time for the web server.  

Any data miner calling to cgi-bin functions should keep the rate of use
in bounds with human usage - no more often than a 3 or 4 times/minute,
unless the server documents allow for more rapid access. The best design
is to pre-test the web functions you want to mine, and see how quickly
they respond.  Then multiply that time by 4 or 5 at least, to allow for
others' use of the server.
*********************************************************

The fbidbatch supports all reports that fbidq does, when you use
an official flybase id of form "FBxx", most commonly gene reports
with FBgn.   

For programmed calls , use these URL query parameters: 
ids=FBgn0000001,FBgn0000123,...   list of IDs 
idclass=FBgn|FBan|FBab|FBrf|FBal|DGC|CG  .. ID class (optional unless DGC or CG) 
format=text/tsv  .. choice of output format 
saveas=file      .. save to file 
fields=ID,NAM,DT .. list of report fields (optional) 
rstyle=Homologs  .. subset report contents (optional) 
  (this is same parameter as 'contents=' in flybase fbidq function).
  
Example URL: 
http://flybase.net/cgi-bin/fbidbatch.html?ids=FBgn0000014,FBgn0000015&fields=ID,GSYM,NAM,DT&format=text/tsv& 
Send comments to us at flybase-help AT morgan.harvard.edu 

by Don Gilbert, Jan 2004.

=cut

# see perldoc LWP
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("FlyBaseDataMiner/101");

my $PleaseBeNiceToServerLoopTime= 60; # one minute delay between repeats

my $debug=1;
  # CGI options for batch ID reports from FlyBase.net 
my $fbidbatchUrl="http://flybase.net/cgi-bin/fbidbatch.html";

 # report format
my $format="text/html"; # also with other params, text/tsv tables, etc. 

 # sub report types - one per call
my @contents=qw(onepage geneinter phenotype);

# idlist, server can handle 10,000s at a time, but you may want to
# iterate over 1,000 per message here to keep client happy.  If you
# iterate calls to fbidbatch, please use a delay between calls of 1+ minute
my @idlist=qw( 
FBgn0000547 FBgn0001987 FBgn0000636 FBgn0001269 FBgn0011592 FBgn0022238
);

# FBgn0005636 FBgn0015300 FBgn0001147 FBgn0036513 FBgn0002932 FBgn0000303
# FBgn0023097 FBgn0039007 FBgn0011666 FBgn0002735 FBgn0000591 FBgn0001139
# FBgn0000150 FBgn0000022 FBgn0000137 FBgn0024367 FBgn0026084 FBgn0002948
# FBgn0030412 FBgn0001079

foreach $contents (@contents) {

  my %params= (
    idclass => FBgn,
    ids => join(',',@idlist),
    format => $format,
    rstyle => $contents,
    );
  
  my $message= join '&', map { "$_=$params{$_}"; }(keys %params) ;
  
  warn "Sending request: $message\n" if $debug;
 
  # Create a request
  my $req = HTTP::Request->new(POST => $fbidbatchUrl);
  $req->content_type('application/x-www-form-urlencoded'); # multipart/form-data
  $req->content($message);
  
  # Pass request to the user agent and get a response back
  my $res = $ua->request($req);
  
  # Check the outcome of the response
  print "\n<br>#--------------- $fbidbatchUrl $contents ---------------<br>\n";
  if ($res->is_success) {
     print $res->content;
  } else {
     print "Bad luck this time\n";
  }
  sleep($PleaseBeNiceToServerLoopTime);
}
