[sword-devel] Creating a "SWORD-over-network" protocol for remote SWORD repo access?

[Aaron Rainbolt]

On Fri, 19 Jul 2024 11:26:09 +0300
Jaak Ristioja <jaak at ristioja.ee> wrote:

...snip...

> > As for the actual issues themselves, on a scale of 1 to 10 how
> > difficult do you think it would be to have a repository descriptor
> > file that is located in a predictable place and that contains data
> > about what modules exist on the server and how to find them? This is
> > more-or-less what the apt package manager does and given Debian and
> > Ubuntu's success it seems to work well.  
> 
> Depends on what you mean by difficulty. I suppose it might only take
> a few working days for the initial design of a simple file format for
> your protocol.

I meant how to solve the issue in libsword itself with finding and
downloading modules. My server shouldn't need a configuration file at
all I don't think.

> >> Another obstacle to defining a new repository format/protocol is
> >> that there is no complete and sound formal specification for the
> >> module configuration file format and its fields. The descriptions
> >> in the SWORD wiki are incomplete and contain ambiguity.  
> > 
> > This is not something that my server idea overcomes, so I'll think
> > about that. Perhaps it would be worth digging into just that and
> > overcoming it by strictly defining the configuration file format?  
> 
> You can use certainly use libsword in the backend, but I don't think 
> that the protocol you're suggesting has to be directly tied to the
> SWORD configuration format. What I mistakingly wrote about was on an
> entirely lower level, and these SWORD specifics might not concern you
> at all. If you want something like this:
> 
>    libsword <-> server <-NEW PROTOCOL-> client
> 
> then it might make sense to ignore everything below the libsword API
> and perhaps also the libsword API itself when designing the new
> protocol. Use the API, but don't inherit it. In this perspective
> perhaps "SWORD-over-network" is a slight misnomer?

Well it's not really a misnomer because I want a graph more like this:

    SWORD repo -> libsword <-> server <-NEW PROTOCOL-> libsword <->
    client 

i.e., libsword would be able to natively support the new protocol so
that clients that wished to use it could do so *almost* transparently.
That way any SWORD client could add network support with minimal
effort, rather than having to be built specifically to use the new
protocol. (The system would still work pretty good even if libsword
doesn't support the protocol and special support is needed, but
probably no existing SWORD clients would pick up support for it.)

> >> While perhaps not strictly be a blocker to creating a new
> >> repository format/protocol, but there are no formal specifications
> >> for the module content and content index files. I remember these
> >> formats having being described as internal libsword details which
> >> don't require specification, because the format and libsword might
> >> change. However, I think this reasoning is incorrect, because
> >> files of these formats are exchanged over the wire, used in
> >> multiple repositories not all which are managed by Crosswire, and
> >> libsword wants to retain backwards compatibility with older
> >> modules as well.  
> > 
> > I agree with you w.r.t. the shortcomings of this. It also makes me
> > realize that it means that the libsword on the server would have to
> > be "close enough" to the libsword of the client in order for my
> > server idea to work, because otherwise the server's libsword will
> > send markup data that the client can't process. If backwards
> > compatibility is still maintained, some way of transferring
> > versioning information over the wire might be enough.  
> 
> I again apologize for the confusion I caused by my reply. I'll try to 
> entangle this.
> 
> When accessing modules the libsword API abstracts away the (outer) 
> format of the content and index files, and presents to the user a way
> to read individual content entries (e.g. verses). So this outer (or 
> container) format might not be of concern to you. The entries
> themselves are fragments of OSIS, ThML, TEI, GBF, plain text etc, but
> the exact formats are also somewhat underspecified. Libsword allows
> you to apply "filters" (transformations) on these entries, including
> ones which convert the entries to other formats, e.g. to (fragments
> of) HTML.
> 
> It is up to the protocol if and where (client or server) any filters
> are applied.

That makes sense.

> >> In my opinion the repository format should not much depend on the
> >> underlying transport protocol (HTTP(S), FTP, local filesystem) and
> >> should not require special handling on the server side. For HTTP
> >> this means that all repository files may be served statically on a
> >> regular web server without requiring extra server-side scripting.
> >> Just files and directories, no parsing of directory indexes, only
> >> retrieval of regular files by their path.  
> > 
> > Hmm, I don't see how this is really possible in a "retrieve part of
> > a module" situation. I mean it probably would work if you used HTTP
> > partial downloads to retrieve the blocks of files you want, but that
> > sounds like it would probably require quite a lot of HTTP requests
> > to load one chapter from a module, which would probably put undue
> > load on the server and slow down the client.  
> 
> Have you thought about the possibility of generating all assets (e.g. 
> entries/verses, and possibly multiple different versions thereof) on
> the server-side statically?

I guess that would be possible but at that point I'll basically have
created a new repository format and be porting SWORD modules to it.

Thanks for the feedback!
Aaron

> Best regards,
> Jaak Ristioja
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page