[sword-devel] Missing ZIP files

Tue Jul 30 14:43:26 MST 2013

Thanks for everyone's input.

Attached is a shell script which is modelled after http://crosswire.org/svn/crosswire-java/trunk/src/org/crosswire/sword/web/PackageMod.java.
PackageMod.java does the following:
Zipping is only done if at least one of the module's files or conf has a more recent modification time than the zip.
The name of the zip is taken from the [NAME] in the conf.
The zip overwrites any existing zip.

My script differs slightly:
It zips to a temporary file in the same directory and then copies it over top of the existing zip.
It manages the mods.d.tar.gz file when a zip is created, again by creating a temporary and moving that over top of the existing.
It assumes that there are no spaces in any path and that the repository is on UNIX.

I've tried this on all the repositories I could find.

I did find that there were a few conf files that did not have module files. Should these confs be renamed, deleted, or otherwise moved out of the way.

I did not update/manage InstallSize, which could be easily added.

Please review and comment.

BTW, it is very verbose.

In Him,
	DM

On Jul 21, 2013, at 5:00 AM, "Troy A. Griffitts" <scribe at crosswire.org> wrote:

> A few quick comments.
> 
> If you're using http to grab the zips, we already have an http URL which will be sure to give you the latest zip-- the packager.
> 
> If I implement the cache mechanism in the engine, I will use the same logic as what is in the packager to determine if any found zip is current.
> 
> The code for the packager is in the crosswire-java svn repo. You should be able to find the servlet with a grep. Old sucky code I'm not proud of but works.
> 
> Chris Little <chrislit at crosswire.org> wrote:
> On 07/20/2013 03:05 PM, DM Smith wrote:
> 
> On Jul 20, 2013, at 5:42 PM, Chris Little <chrislit at crosswire.org> wrote:
> 
> On 07/20/2013 07:14 AM, Martin Denham wrote:
> They should be there again now.  And Bible depends on the av repository.
> 
> See this thread for more info:
> http://svn.crosswire.org/pipermail/jsword-devel/2013-April/004639.html
> 
> It is a real shame the av zips are not created automatically.
> 
> 
> Or...
> It's a real shame that JSword doesn't use the repositories correctly, according to their intended usage.
> 
> Yes it is a shame. I'm almost done w the code to use the expanded repository via ftp. This will be with the release after next.
> 
> Awesome.
> 
> Or...
> It's a real shame you don't write a cron job that can update the ZIPs nightly, if necessary. (Maybe Karl can help you, if he has a script to update ZIPs in his repository.)
> 
> Can anyone point me to the code that does the zipping? While coding it up is trivial, there's no point to re-inventing the wheel. Just port the code to a loop.
> 
> The latter is probably more useful at this point and more urgent since I am going to write a cron job to delete the ZIPs on occasion.
> 
> Please don't do this until we have had a chance to write the script.
> 
> Yeah, it's not urgent. It can wait a while. And I only intend to have it 
> run once a week or so.
> 
> What would the purpose of doing the mass deletes? Currently JSword uses the file time to determine whether the module has changed. Since our frontends do not advertise updated modules, this is no big deal. We've got the mechanism written to use the conf version field, but it is not in the frontends yet.
> 
> There are a couple of reasons to clear out the cache on occasion. The 
> web download applet will generate ZIPs for updates and new modules, but 
> don't remove pulled modules (and sometimes I forget to do it manually). 
> We also get people who think they're really smart and start feeding the 
> download applet things like NIV and NKJV. And the applet will happily 
> package those for them (except that they are empty of content). So it's 
> nice to clear out those sorts of files from the server.
> 
> If the purpose is to ensure that old stuff is appropriately deleted, then a script comparing module conf to zips deleting those that don't match would be better. This is even simpler to write than the zipper program. This could be added to the zipper program.
> 
> Sure, there are other ways of handling the issues.
> 
> BTW, Troy's position has been that it is OK to go to the zip cache first and failing that get the module by parts. But that there should be no requirement of a frontend to have the zips. Many have noted that getting the zip is faster and more reliable than getting by parts.
> 
> At present, I believe that is a bad position. Shortly after updates, 
> ZIPs and the module parts can get out of sync so that, e.g., you think 
> you're downloading version 2.0 of a module, but you're actually 
> downloading 1.0 since that's what the ZIP is because no one has 
> downloaded through the applet, causing the ZIP to update. And no amount 
> to trying to update from the ZIP is going to update to 2.0 until someone 
> else hits the download applet.
> 
> A cron job running an update script would at least mean that both the 
> mods.d.tar.gz and ZIPs get updated at approximately the same time.
> 
> --Chris
> 
> 
> 
> 
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
> 
> -- 
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20130730/6bd3e06c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: packageModules.sh
Type: application/octet-stream
Size: 4329 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20130730/6bd3e06c/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20130730/6bd3e06c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4145 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20130730/6bd3e06c/attachment.p7s>