[sword-devel] Missing ZIP files

[Chris Little]

On 07/20/2013 03:05 PM, DM Smith wrote:
>
> On Jul 20, 2013, at 5:42 PM, Chris Little <chrislit at crosswire.org> wrote:
>
>> On 07/20/2013 07:14 AM, Martin Denham wrote:
>>> They should be there again now.  And Bible depends on the av repository.
>>>
>>> See this thread for more info:
>>> http://svn.crosswire.org/pipermail/jsword-devel/2013-April/004639.html
>>>
>>> It is a real shame the av zips are not created automatically.
>>
>>
>> Or...
>> It's a real shame that JSword doesn't use the repositories correctly, according to their intended usage.
>
> Yes it is a shame. I'm almost done w the code to use the expanded repository via ftp. This will be with the release after next.

Awesome.

>> Or...
>> It's a real shame you don't write a cron job that can update the ZIPs nightly, if necessary. (Maybe Karl can help you, if he has a script to update ZIPs in his repository.)
>
> Can anyone point me to the code that does the zipping? While coding it up is trivial, there's no point to re-inventing the wheel. Just port the code to a loop.
>>
>> The latter is probably more useful at this point and more urgent since I am going to write a cron job to delete the ZIPs on occasion.
>
> Please don't do this until we have had a chance to write the script.

Yeah, it's not urgent. It can wait a while. And I only intend to have it 
run once a week or so.

> What would the purpose of doing the mass deletes? Currently JSword uses the file time to determine whether the module has changed. Since our frontends do not advertise updated modules, this is no big deal. We've got the mechanism written to use the conf version field, but it is not in the frontends yet.

There are a couple of reasons to clear out the cache on occasion. The 
web download applet will generate ZIPs for updates and new modules, but 
don't remove pulled modules (and sometimes I forget to do it manually). 
We also get people who think they're really smart and start feeding the 
download applet things like NIV and NKJV. And the applet will happily 
package those for them (except that they are empty of content). So it's 
nice to clear out those sorts of files from the server.

> If the purpose is to ensure that old stuff is appropriately deleted, then a script comparing module conf to zips deleting those that don't match would be better. This is even simpler to write than the zipper program. This could be added to the zipper program.

Sure, there are other ways of handling the issues.

> BTW, Troy's position has been that it is OK to go to the zip cache first and failing that get the module by parts. But that there should be no requirement of a frontend to have the zips. Many have noted that getting the zip is faster and more reliable than getting by parts.

At present, I believe that is a bad position. Shortly after updates, 
ZIPs and the module parts can get out of sync so that, e.g., you think 
you're downloading version 2.0 of a module, but you're actually 
downloading 1.0 since that's what the ZIP is because no one has 
downloaded through the applet, causing the ZIP to update. And no amount 
to trying to update from the ZIP is going to update to 2.0 until someone 
else hits the download applet.

A cron job running an update script would at least mean that both the 
mods.d.tar.gz and ZIPs get updated at approximately the same time.

--Chris