[sword-devel] SWIG Bindings (Was: Re: What is a Sword module?)

[Ben Morgan]

Hi Greg,

On Sat, Nov 6, 2010 at 12:24 PM, Greg Hellings <greg.hellings at gmail.com>wrote:
>
> But the nature of multimap is different - and I'm confused as to how
> it is used at all.  What you are showing above I would call a
> multi-dimensional associative array/dictionary.  I make them all the
> time in Python and PHP.  But a multimap is defined as "a
> generalization of a map or associative array abstract data type in
> which more than one value may be associated with and returned for a
> given key."  I don't think we are actually using multimap
> functionality but rather just a multidimensional array.
>
If I remember rightly, a multimap is just like a dictionary in python that
can have multiple values for a given key (conceptually I guess it is like a
dictionary of lists of something).
The getEntryAttributes stuff is not a multimap - just nested maps.

Multimaps are seen in SWConfig, where it is possible to have multiple
entries for a given key (e.g. in modules, multiple GlobalOptionFilter's).

This allows you to iterate through them like this:
(annoyingly long fragment adapted from C++, untouched for a year
probably...)

# look first in the map
sourcesSection = installConf.getSections().find(Sword.SWBuf("Sources")) if
sourcesSection != installConf.getSections().end(): ftp_source =
Sword.SWBuf("FTPSource") ss = sourcesSection.value()[1]
# this is where we start iterating over the multimap sourceBegin =
ss.lower_bound(ftp_source) sourceEnd = ss.upper_bound(ftp_source) while
sourceBegin != sourceEnd: install_source = InstallSource("FTP",
sourceBegin.value()[1].c_str()) print privatePath + "/" +
install_source.source
sourceBegin += 1



> SWIG gives us multimap support in Python for free and I found a
> version we include with the library to get Perl working.  However,
> SWIG does not have an implementation of stl::multimap for us, so we
> either need to write one, or refactor our multimap usage to be nested
> associative arrays.
>
 Most people wouldn't need to touch multimap - I think it's only used  in
SWConfig, where mostly you can just use get and set on the SWConfig if
needed (only if you need to iterate over multiple key values). So I don't
think this should worry us too much. To some extent, if an area of the API
isn't supported for a given language, there's no point in supporting it
until someone needs it.


> >
> > But this interface doesn't translate well to other languages.  I have
> > had to implement a different interface for my custom language binding
> > for Java-jni (used by SWORD C++ on Android) and CORBA (used by SWORDWeb).
> >
> > The binding interface I usually implement is described in simple IDL
> > here and is a fairly concise set of methods that give access to most all
> > of functionality of the engine:
> >
> > http://crosswire.org/svn/sword/trunk/bindings/corba/swordorb.idl
> >
> > You'll notice the method for obtaining entry attributes:
> >
> > StringList SWModule.getEntryAttribute(in string level1, in string
> > level2, in string level3, in boolean filtered);
> >
> > This less exotic interface maps much easier to other languages, but
> > implies you know what attribute you'd like to obtain (discovery by
> > iteration is not supported, but really, what frontend does that anyway?)
>
That might be a better way of doing it, and would be very easy to add.
In BPBible we don't exactly iterate over it, but we do get values at
different levels and check whether subvalues exist (look at:
http://code.google.com/p/bpbible/source/browse/trunk/backend/book.py#415
). This we could replace with a simpler interface in getEntryAttribute
(which could return None if key not found).

I can imagine a situation where someone wanted to do iteration, but we don't
seem to do that in BPBible at the moment...

>
> > So, in summary, maybe we should add this to swmodule.i
>
> That is one possibility.  However, I think there are other places
> within the bindings that should be addressed first.  As you saw
> yourself, SWBuf plays great in C++ but SWIG doesn't seem to pick up
> any ability to auto-cast it back to a Python string/unicode object.
> At the very least, that functionality should be as easy as an explicit
> cast of unicode(myswbuf) and better yet it would be transparent.
>
I tried doing that with a simple SWIG typemap, but it didn't play well with
the STL stuff. Almost everything worked as expected, but when a SWBuf was a
key in a std::map for example, it didn't transparently convert that, and
there was no way to create SWBuf's then.
Occasionally it is useful not to transparently convert them but to look at
the length or something like that for efficiency (not very common I
think...)

Depending on how smart you want to try to make it, SWBuf's have to end up as
byte strings as well quite a bit of the time. There's no universal way from
the API to tell whether something is utf-8 or cp1252. You could do it for a
number of key interfaces, though - RenderText, getKeyText, get config entry
from module.

Ideally, someone would write extra Python code so that the SWIG
> bindings were wrapped in a more naturally Pythonic binding.  It might
> be perfectly natural in a language like C++ or Java to access a module
> with
>
> SWMgr* mgr = new SWMgr(FMT_HTMLHREF);
> SWModule* mod = mgr->getModule("KJV");
> VerseKey* key = (VerseKey*) mod->getKey();
> key->persist(true);
> key = 'john 3:1';
> cout << mod->RenderText();
>
> A natural pythonic way of doing this would probably look more like
>
> mod = Sword.module('KJV')
> print mod('john 3:1', format='HTMLHREF')
>
You could easily put a simple python wrapper around Sword which did this,
but this wouldn't easily cover everything. From my perspective:
1) SWMgr is very necessary to expose for BPBible at least. Probably not in a
simple API though.
2) I don't like that function style calling, and I wouldn't say it is
pythonic (not that I necessarily am a good judge of that). A method would
work better I think - say text or get_text, but taking a string.
3) Passing in format like you did won't (easily) work as the format is a
property of the SWMgr. Unless you keep a SWMgr per format, and look them up
and create them when needed. Probably defaulting to HTMLHREF is correct, and
then having a module level set format to call which creates a new SWMgr.

Thos problems aside, a simple python wrapper could create a much lower
barrier to entry. Personally, I don't care too much about other languages
than Python.

To some extent, I've simplified working with some of the SWORD stuff and
done extra checking and whatnot in BPBible, mostly in the file below:
http://code.google.com/p/bpbible/source/browse/trunk/swlib/pysw.py

This file does quite a bit, mostly with (Verse, Tree, SW, List)Keys. Keys
will be the hardest thing to make work nicely, I think. In BPBible we do
lots of special stuff with keys, for example formatting ranges nicely,
keeping user input segregated from english keys, etc.
The stuff in this file is reasonably tied into BPBible, and so isn't usable
as-is elsewhere.

An example of how this could look:
mod = Sword.module("KJV")
print mod.get_text("John 3:16")
print mod.get_text("John 3")
for verse, text in mod.chapter("John 3"):
    print verse, text

for reference, text in mod.range("John 3:1-6"):
    print reference, text


> But in order to get to that point would take someone writing
> native-feeling Pythonic bindings and making design decisions like
> that.  I'm tied up with other tasks at the moment and can't devote the
> time to learning SWIG to clean up the warnings and errors the bindings
> throw on compile PLUS write native feeling bindings for languages.
> But it'd be a great thing for someone to tackle if they really like
> their scripting languages and probably wouldn't require learning much
> (if any) C++.
>
Trying to work with SWIG at all is likely to end up with using C++. If doing
any stuff with stl, it is really needed, and even still very hard to get
working (I did all the stl map/vector stuff, and it was painful and
fragile). A simple wrapper API in python would require understanding SWORD,
but possibly not too much C++ unless more SWIG binding needed writing.

As to the warnings from SWIG, some I tried to set SWIG to ignore, but it
wouldn't. So I gave up. Others are possibly valid concerns, but I wasn't
sure how to handle them. Writing SWIG bindings is painful, in case anyone
didn't realise... :-)

The bottom line is that SWORD is a complex API, but for a full-featured
frontend, a lot of that complexity is necessary.

God Bless,
Ben
-------------------------------------------------------------------------------------------
Multitudes, multitudes,
    in the valley of decision!
For the day of the LORD is near
    in the valley of decision.

Giôên 3:14 (ESV)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/8907848c/attachment.html>