[sword-devel] New public git mirror of Sword SVN trunk and why

[Troy A. Griffitts]

Dear Jaak,

I appreciate the heart of your last two emails.  Collaboration has 
always been at the core of our purpose for the existence of CrossWire 
and I would love nothing more than to share together in work with you.

Before I answer specifics in your emails, may I please suggest that you 
try to enter our developer community by seeing a very small and 
insignificant need, discuss a fix and receiving feedback on the best way 
to go about that fix, and then submit a patch.  I think you will find 
that we are eager to accept quality contributions to the code and would 
be excited to have an additional contributor.  I would be overjoyed to 
have someone help.


On 12/18/2012 01:03 AM, Jaak Ristioja wrote:
> As developers of a front-end, we are keen to have new features added.

Yes, me too!  This is where to discuss them.

> We're displaying the module texts as HTML. But not just as Sword 
> passes them to us, but we would like something more. We want to 
> transform those Sword outputs, e.g. add some interactive features etc 
> etc. This requires some sort of additional processing. Developing such 
> processing is complicated, because we're not absolutely sure about the 
> format of the output Sword produces. Sometimes it has not been valid 
> and we've seen strange markup being output to the user which he or she 
> should not see. Working around such things has been a pain. Sword 
> lacking a full formal specialization (e.g. BNF grammar) of the output 
> in its documentation is a problem for us.

Let me start by saying that the process you describe here is 
challenging:  taking complex tagged texts in various markup formats, 
submitted by a diverse group of content authors, and transforming each 
of these into something valid and consistent.

This is our goal, and the grail of the engine.

All I can say is that, as we encounter new and creative methods of 
markup, we do our best to normalize these into something sane and try 
our best to balance between:
1) preserving intended features of the author, with
2) normalizing as much as possible to conform the output to some common 
markup

I get slammed on both sides.  Content authors and their representatives 
here want me to touch as little as possible. Frontend developers cry for 
a consistent output.


> Of course I don't understand what's under the hood. I've been studying 
> the code and any documentation I've found, but still haven't figured 
> it out. 

I'm happy to help if you have questions.  If you'd like to contribute to 
ease the pain for the next engineer, as you go through this learning 
process, please please please diagram or document concepts as you 
learn.  I would love for a contributor to fill this void.


> Sword could apply a workflow which would not 'churn' the code, e.g. 
> with gatekeepers who optionally merge changes from others repository 
> clones to the main repository. See 
> http://git-scm.com/book/en/Distributed-Git-Distributed-Workflows 
> http://thkoch2001.github.com/whygitisbetter/#any-workflow 

We have a well documented development workflow we advertise and hold as 
our goal.  It is the very first link ("Development Process") on our 
developer page on the The SWORD Project website:

http://crosswire.org/sword/develop

It is far from reality and you can see from the wording (e.g., "CVS") 
that the tools have changed, but the *intended* workflow is still our 
desire.


> First of all, being rebellious is not my intent.
Thank you.  Your subsequent emails were very helpful to diffuse this 
impression.

> I just want to push the project for some good synergy. In reply to my 
> first post I have already received personal emails with requests to 
> actually start handling a fork, do a release from it ASAP
I would love for someone to hold the "release manager" pumpkin, to 
schedule dates and herd developers to get their fixes in before a 
deadline, call for testing periods, and then package up official 
releases.  Maybe someone out there loves doing this.  I don't.  I'm bad 
at it.  Great place for someone to step in and make an immediate impact.


> , fix the interfaces

We've been slowly normalizing interfaces toward a 2.0 release for years 
now, but this involves mostly naming conventions.  There are always 
improvements we could do, but please realize any change to an interface 
impacts many people.  Without a diatribe on proper interface design, is 
there one single small interface 'bug' you've specifically had trouble 
with, which you honestly feel warrants your time and energy to suggest a 
change, and would warrant many client projects' change to adapt to that 
interface change?  If so, I'm happy to entertain a constructive 
conversation about one specific problem you've had and your suggested 
"interface fix".  Please start a new thread, if so.


> and compiler warnings.

We've had arguments about the level of catering we will do for compiler 
warnings.  Different distros turn up checks and warnings to different 
levels.

The current place we are at here is that I have conceded to use -Wall 
-Werror (turn on all warnings; count warnings as errors) while we are in 
development mode.  We try to keep up with any newly added warning.  Some 
distros also force CFLAGS to enable paranoid levels of warnings.  e.g.,

https://wiki.kubuntu.org/ToolChain/CompilerFlags?action=show&redirect=CompilerFlags

We do not count these as errors.  It's fine to see what these say and 
consult them periodically to review possible problems in code, but these 
often require code to be changed, simply to cater to removing the 
warning and make the code less understandable.  Again, I'm not saying 
that these checks are not valuable; I'm simply saying that we don't 
count everything output from these extra check as errors.


> But we need fixes to applied faster, bugfix releases to be made 
> earlier etc.

I'm not aware of acceptable fixes that are being held for an 
unacceptable length of time out of SVN (well, not longer than levels of 
delinquency that fit within the realm of our opensource world).

We do have a serious release deficiency.  Any help here would be greatly 
appreciated.


> Me and other volunteers are willing to do some work for this. We're 
> asking for Sword to meet us halfway.

Excited. Very much welcomed.

> I work as a C/C++ engineer, I have a MSc in Computer Science, 
> specializing in programming language theory. At work, I refactor a lot 
> of code, I read the ANSI C and C++03/C++11 standards (drafts) almost 
> daily.

I appreciate your skillset and welcome your talents.


> I'd like to extensively refactor and optimize your code.

I am naturally a conservative and this raises immediate alarms.  We have 
some very heavily optimized code.  So much so that we use our own string 
class (SWBuf) which, on most platforms, is faster than most C++ STL 
implementations of std::string


> Are you sure Sword could handle my stream of patches by email? 1) 
> Anyway, what is the process of submitting patches? Only by email? git? 
> 2) What is the process of reviewing and merging patches? 3) Who does 
> all that? 4) How do I monitor this process? 5) What are my options to 
> interact and accelerate development? Maybe I've missed some 
> documentation, but generally I feel that these are unanswered 
> questions. This is one obstacle. How does one contribute? 
We have a core group of engineers who have full write access to our 
authoritative repository.  They have gained this right by displaying 
respect for our objectives and the developers who have submitted code 
before them.  They change others' code very infrequently-- only out of 
necessity, and always defer to pumpkin holders of individual peaces in 
the code base.  This may sound arrogant or controlling, as I hold the 
pumpkin for most of the engine, but we do have many people responsible 
for other parts.  DM (probably the bane of his existence) holds the 
pumpkin for osis2mod.  Xiphos (though they probably wouldn't claim it) 
hold the pumpkin (at least I defer to them always when I change things 
there) for the HTMLHREF filters, though it sounds like they've recently 
moved to the XHTML filters (which I'm excited about and hope more 
frontends can share this filter set).  Chris holds the pumpkin for the 
SWFilterMgr and many of the filters-- particularly the ones dealing with 
encoding or Unicode code points.  Greg, the CMAKE system.  Others, the 
bindings (not sure who's still claiming which bindings these days).  
Many people come and go and I inherit things as they leave until someone 
else takes up their cause (autoconf build system, compressed drivers (z* 
classes), etc).

There are many places you would be met with a warm welcome to contribute 
and improve, but please understand the defence we feel when someone 
comes along and says they want to refactor and optimize our code.

On 18.12.2012 07:11, Troy A. Griffitts wrote:
>> I have been following the discussion on the SFTP patch and hadn't
>> seen it come to a conclusion yet regarding what might be necessary
>> to detect SSL support in cURL. I don't feel I've been negligent
>> with this.
> How about allowing any custom transports layers to used via the
> library interface?

We try to make things easy to allow customization of transports. 
Bibletime, in fact, replaces our default implementations (or at least 
did at one time) with Qt implementations of transports.  It is not 
difficult to add recognition of an entirely new transport (see Greg's 
patch), but could be easier if we felt new transports were being added 
with any regularity.  We do have ways to 'register' handlers in the 
engine for many things, but this pattern hasn't been fully realized to 
ease the work necessary for new types of transports, as there hasn't 
been much of a need over the years.

> We must all keep in mind that we share common goals. We are discussing
> these things to try initiate processes to help our projects and the
> community. There are issues and I pray that God would help us to solve
> these in His love and understanding and in a brotherly fashion.
Thanks Jaak.  Please know I too get frustrated with the slow process of 
development.  I would love for people to come along side and share in 
the effort-- especially doing things I hate to do.  But-- no offence 
meant toward you-- we often receive more whining about deficiencies and 
what isn't getting done, than people willing to settle in and contribute 
hard time to pick up where things are sagging.  I say I'll do things for 
other people and often don't get them done in a timely fashion.  That's 
no fun for me or for those asking for the changes.  It would be great to 
have more champions around who could take up requests and submit quality 
solutions for the deficiencies.  I sincerely hope you become one of those.

Sincerely anticipating brotherly co-laboring,

Troy