[server-admins] makeDownloadsStats.pl

Nic Carter niccarter at mac.com
Thu Mar 18 02:33:17 MST 2010 

Ok, well, the ~sword/bin/makeDownloadsStats.pl script will process:

/var/log/httpd/crosswire-access_log
and
/var/log/xferlog

After digging a little, it seems like because I access http://ftp.crosswire.org via the SWORD HTTP install manager, it is defaulting to the first (alphabetically) listed VirtualHost in apache, which is the community one.  Hence it's getting logged there.

There are 2 options.  And, actually, we could do both?

1) modify our stats script to also work for the community log files
2) modify PocketSword to start logging to the crosswire log files
3) modify the apache conf so that there is a virtual host for ftp (which is the same as the crosswire one)

ok, so I can't count.  :P

my comments:

1) useful, as it will mean running the script will include stats from Dec - March from PocketSword.
2) I can do this, but it won't fix the Dec-March issue.  Also, it won't take effect until a new version is released by Apple.
3) Easy, but won't fix the Dec-March issue.  Will take effect instantly (unlike option 2).

Let me know if you want me to do any or all of the above?  My personal preference would be for us to do (1) & (3).  I prefer (3) to (2) simply because it will take effect instantly. :)

Actually, I've attached an updated stats script.  Not much has changed, but I've tested it and it works as much as I can tell (I can't access the ftp logs, cause for some reason they're not world readable -- perhaps it logs ppl's passwords, hence not wise to be 644?).

NOTE:  If anyone else was using the C++ SWORD HTTP install manager, those downloads would have not been shown in our current script, as the script was simply looking for HTTP downloads of ZIP files, whereas the install manager will be downloading the individual files in each module.  Hence, my point #1 above is actually not an option, it's required.  do a diff on my attached script and the current one and you'll see my updated code to take this into account.  I had actually completely forgotten about this, as I think I wrote this code at the start of January?

FYI, while we're talking about download stats, theoretically our KJV and MHCC and Robinsons will be a little out, because PocketSword bundles these (along with the Xiphos versions of the 2 Strong's Numbers modules).  So to accurately determine the number of copies of those modules that are being downloaded (from both CrossWire and Apple), we would need to add in the PocketSword download stats.  (Robinsons and Strongs will be bundled with v1.2, which is yet to be released).
But, then again, I'm sure this is the same with our other front ends!  :)

I hope all of that helps  :)


Thanks, ybic
	nic...  :)


On 18/03/2010, at 7:19 PM, Troy A. Griffitts wrote:

> Maybe I don't understand.  Why do you think that http downloads via the
> API are any different than browser web downloads?  I really can't read
> Perl and am told most people who write Perl can't read Perl either, but
> I'm not sure the API accessing the website is any different than an end
> user accessing the website via a browser.
> 
> I don't know who owns:
> 
> ~sword/bin/makeDownloadsStats.pl these days.  Can someone comment who
> has a history with the script?
> 
> Nic Carter wrote:
>> That is correct, and the reason for them not appearing there is because those stats don't take into account downloads using the HTTP version of the SWORD install manager.
>> 
>> Remember how that has only recently come into being?
>> 
>> So, the ONLY reason the stats don't appear there is because we haven't updated our script to include all 3 of (we currently only do 2 of these, I believe???):
>> + FTP install manager (already there)
>> + direct web http downloads (I believe is already there, but I admit that I haven't looked too closely to see if these stats are there)
>> - HTTP install manager (what my new script does, but I'm fairly certain isn't currently in any of the old scripts).
>> 
>> I'm really sorry I didn't point this out earlier, I was meaning to post to sword-devel about this, but I didn't realise that it was important enough to have emailed straight away?  :(
>> 
>> Please let me know what I can do.
>> 
>> On 18/03/2010, at 7:06 PM, Troy A. Griffitts wrote:
>> 
>>> Nic,
>>> 
>>> This is potentially bad news.  I just finished a long email to Dan about
>>> how your downloads are exactly the same as any other CrossWire download
>>> and gets logged in the stats we send Crossway.  Are you saying that the
>>> numbers on our front page generated by makeDownloadsStats.pl
>>> 
>>> http://crosswire.org/sword
>>> 
>>> on the left side, for the ESV, do NOT include the stats for PocketSword
>>> installs?
>>> 
>>> 
>>> 
>>> Nic Carter wrote:
>>>> Hi there Troy & Chris,
>>>> 
>>>> In regards to the email I cc'd to you earlier today (to Daniel at Crossway) (this email to sword-support jogged my memory!), when users download modules through PocketSword, that download gets logged differently to a normal FTP or web download.  Not long after PocketSword went live, I looked into this on the webserver & discovered that the downloads are being logged in /var/log/httpd/community-access_log which is great for me, cause it means I can see easily & quickly which module downloads are by my users.  In fact, if you look at that access log, you can grep for "installmgr at pocketsword.iphone.com" and find all the GET requests from PocketSword.  I simply picked a nice and obvious name so that we'd know at a glance which are from PocketSword.  To do this, I edited curlhttpt.cpp (added a line for "curl_easy_setopt(session, CURLOPT_USERAGENT, p.c_str());" & I have set the password I use to be that string) and I'm gonna have to keep my copy of SWORD up-to-date in order to 
> al
>>> low for this.  I only added the "installmgr at pocketsword.iphone.com" string in v1.1 of PocketSword, so it won't show all the PocketSword module installs.  However, there didn't seem to be any other requests for modules that are logged in this log file, so I was able to easily and quickly edit our makeDownloadsStats.pl and generate stats for the downloads.
>>>> If you look in ~niccarter/local/bin you will find my hacked script & stats that I generated about 10 mins ago, showing that the ESV has been downloaded 1990 times by PocketSword users since it was released in December last year.
>>>> 
>>>> Anyway, I'm praying that the email that I sent to Daniel is able to clear things up.  If required, and as a last resort, I can manually put in a filter for the ESV from our repository so that it doesn't appear in the downloads tab in PocketSword.  That would be preferable to Crossway saying we can't distribute the ESV anymore!  It would mean that if someone wanted to install the ESV into PocketSword in the future, they'd have to download the ZIP of the ESV from our website and then manually install it using the manual install method I've built into PocketSword.
>>>> 
>>>> I'd love to hear back from either of you about all this!  I really hate treading on toes, and really hope I haven't with all this.
>>>> 
>>>> Thanks, ybic
>>>> 	nic...  :)
>>>> 
>>>> ps:  I'm not sure who sent the email from feedback at crosswire.org, but I'm assuming it was Troy.  Given that I think of Chris as the "modules guy", I've cc'd this to Chris as well  :)
>>>> 
>>>> ----
>>>> Nic Carter
>>>> PocketSword Developer - an iPhone Bible Study app
>>>> www: http://crosswire.org/pocketsword
>>>> iTunes: http://itunes.apple.com/app/Pocketsword/id341046078
>>>> Twitter: @PocketSword
>>>> 
>>>> Begin forwarded message:
>>>> 
>>>>> From: CrossWire Bible Society <feedback at crosswire.org>
>>>>> Date: 18 March 2010 6:36:26 PM AEDT
>>>>> To: dkok at crossway.org
>>>>> Cc: sword-support at crosswire.org
>>>>> Subject: [sword-support] ESV 2009-2 download stats from CrossWire
>>>>> Reply-To: SWORD Support Volunteers <sword-support at crosswire.org>, feedback at crosswire.org
>>>>> 
>>>>> ESV:	5093
>>>>> 
>>>>> _______________________________________________
>>>>> sword-support mailing list
>>>>> sword-support at crosswire.org
>> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: makeDownloadsStats.pl
Type: text/x-perl-script
Size: 14155 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/server-admins/attachments/20100318/4fdac384/attachment.bin>
-------------- next part --------------

More information about the server-admins mailing list