[jsword-devel] Concurrent Test failure?
Chris Burrell
chris at burrell.me.uk
Mon Feb 10 04:59:26 MST 2014
As far as I can see Cloudbees is failing on a different test now in
RawFileBackendTest. I believe Cloudbees doesn't have access to /tmp, so
trying a workaround... Or something has changed in the backends to force
the presence of OT and NT files (Possibly my changes 2 years ago!)
Either way, the JUnit 3 remnant was causing the @Ignore to be ignored! Pull
request on github
Chris
On 8 February 2014 16:51, DM Smith <dmsmith at crosswire.org> wrote:
> CloudBees is now having the same failure.
>
> The @Ignore flag is on it. I have no idea what that does. It does not
> cause the test to be ignored when I run the tests via ant using JUnit 4.
> I'm not up to speed on JUnit 4 or annotations at all. I'm still mentally
> stuck on JUnit 3, having not bothered to read about the upgrade.
>
> DM
>
> On Feb 2, 2014, at 3:15 AM, Chris Burrell <chris at burrell.me.uk> wrote:
>
> Interesting. I use Java 7 and it's not an issue (builds both on my windows
> laptop and our build server running linux). Maybe it's because my code base
> has the @Ignore flag on them. Does yours not?
>
> If the create is necessary for other tests, it feels like it should be
> done in the @Before method, or @BeforeClass
>
> Chris
>
>
>
> On 2 February 2014 00:54, DM Smith <dmsmith at crosswire.org> wrote:
>
>> I'm having test failures in RawFileBackendTest. It appears that JUnit 3
>> always did tests in declaration order. While JUnit 4 does not define that.
>>
>> The problem is that the RawFileBackendTest does a setup and then the
>> first test used to be testCreate, which would create the module. The next
>> two tests then used the result of that creation.
>>
>> When I run with Java 7 testCreate is executed last. This cause the other
>> two tests to fail in backend.initState() because the module does not exist.
>>
>> Does anyone know JUnit 4 well enough to:
>> a) Have a test that merely tests module creation.
>> b) Executes that test before tests of that module.
>>
>> That is forces the old JUnit 3 declaration order of execution or the
>> equivalent.
>>
>> When I've googled it there are all kinds of condescending "how stupid can
>> you be to have dependency order between tests."
>>
>> It seems that a test should do one thing and do it well. Obviously, the
>> test class can be rewritten to have a single test that does it all. Then it
>> is not an issue.
>>
>> In Him,
>> DM
>>
>> On Feb 1, 2014, at 6:20 AM, Chris Burrell <chris at burrell.me.uk> wrote:
>>
>> This should now be fixed. Can someone re-test please?
>>
>> I tested with an empty repo and everything worked fine.
>> Chris
>>
>>
>>
>> On 20 January 2014 16:38, DM Smith <dmsmith at crosswire.org> wrote:
>>
>>> We hope to get there. I'm no maven expert so Chris or Douglas "qmx" will
>>> be doing it.
>>>
>>> -- DM
>>>
>>> On Jan 20, 2014, at 11:35 AM, Greg Hellings <greg.hellings at gmail.com>
>>> wrote:
>>>
>>> If this gets resolved, would it then be possible for JSword to be hosted
>>> in the public Maven repositories? It would make building and deploying
>>> against JSword that much easier!
>>>
>>> --Greg
>>>
>>>
>>> On Mon, Jan 20, 2014 at 10:33 AM, DM Smith <dmsmith at crosswire.org>wrote:
>>>
>>>> Please do!
>>>>
>>>> I had thought that the problem was that Maven was executing tests in a
>>>> different order and that it always started with a clean slate. I just
>>>> checked and the tests do not install the KJV or ESV. So that is needed and
>>>> it needs to be forced to be early.
>>>>
>>>> Since I maintain both the KJV and ESV, they are always in my local
>>>> install.
>>>>
>>>> In Him,
>>>> DM
>>>>
>>>> On Jan 20, 2014, at 11:21 AM, Chris Burrell <chris at burrell.me.uk>
>>>> wrote:
>>>>
>>>> Apologies I hadn't read the whole thing. If we can include that test,
>>>> then that would make it work.
>>>>
>>>> Just add something like this in the plugins section in Maven;
>>>>
>>>> <plugin>
>>>>
>>>>
>>>>
>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>
>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>> <executions>
>>>>
>>>>
>>>>
>>>> <execution>
>>>> <configuration>
>>>>
>>>>
>>>>
>>>> <includes>
>>>> <include>**/BookInstallerTest.java</include>
>>>>
>>>>
>>>>
>>>> </includes>
>>>> </configuration>
>>>>
>>>>
>>>>
>>>> <phase>process-test-classes</phase>
>>>>
>>>> <goals>
>>>> <goal>test</goal>
>>>>
>>>>
>>>>
>>>> </goals>
>>>> </execution>
>>>> </executions>
>>>> </plugin>
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>> On 20 January 2014 16:19, Chris Burrell <chris at burrell.me.uk> wrote:
>>>>
>>>>> Indeed I think that's the issue. With STEP, as part of the Maven
>>>>> process we install the modules automatically.
>>>>>
>>>>>
>>>>> https://github.com/tyndale/step/blob/master/step-core/src/test/java/com/tyndalehouse/step/core/prebuild/DownloadJSwordBiblesPreReq.java
>>>>>
>>>>> The above file is what I use to ensure the tests pass regardless...
>>>>> With a Maven snippet to ensure that test always run before every other
>>>>> test.
>>>>>
>>>>> It uses the STEP service-orientated approach, so you'd have to rewrite
>>>>> bits of bit to make it work for JSword - I suspect it's worth it in the
>>>>> long run however. (Wouldn't take long I suspect)
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 20 January 2014 16:13, Martin Denham <mjdenham at gmail.com> wrote:
>>>>>
>>>>>> I suppose CloudBees does not have any modules installed? The errors
>>>>>> it reports look similar to the errors I had just now when I didn't have any
>>>>>> Books installed on my local drive, before I installed KJV and ESV.
>>>>>>
>>>>>> The junits do not seem to install any modules but assume kjv and esv
>>>>>> are installed. I added the following to my local JSword and ran it before
>>>>>> the junits and then they all ran fine. Maybe the fix would be to tell
>>>>>> maven to only include<https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html>AllTests.java and to add BookInstallerTest (below) as the first test in
>>>>>> AllTests, but I am no expert with maven or CloudBees:
>>>>>>
>>>>>> public class BookInstallerTest extends TestCase {
>>>>>>
>>>>>> private BookInstaller underTest = new BookInstaller();
>>>>>>
>>>>>> @Test
>>>>>> public void testInstallBook() {
>>>>>> try {
>>>>>> if (Books.installed().getBook("KJV")==null) {
>>>>>> Book kjv = underTest.getRepositoryBook("CrossWire",
>>>>>> "KJV");
>>>>>> underTest.installBook("CrossWire", kjv);
>>>>>> }
>>>>>> if (Books.installed().getBook("ESV")==null) {
>>>>>> Book esv = underTest.getRepositoryBook("CrossWire",
>>>>>> "ESV");
>>>>>> underTest.installBook("CrossWire", esv);
>>>>>> }
>>>>>> } catch (Exception e) {
>>>>>> e.printStackTrace();
>>>>>> fail(e.getMessage());
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>>
>>>>>> On 29 October 2013 21:39, Chris Burrell <chris at burrell.me.uk> wrote:
>>>>>>
>>>>>>> I suspect names[i] refers to a module you don't have.
>>>>>>>
>>>>>>> Step's maven build has something which automatically downloads
>>>>>>> missing modules before tests are run. Perhaps we can port that across?
>>>>>>>
>>>>>>> Chris
>>>>>>> On 29 Oct 2013 16:13, "Greg Hellings" <greg.hellings at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The specific failures I'm seeing are:
>>>>>>>> Running org.crosswire.jsword.book.ConcurrencyTest
>>>>>>>> Exception in thread "Thread-3" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-5" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-7" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-9" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-11" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-13" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-15" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-17" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-19" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-21" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-23" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-25" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-27" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-29" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-31" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-33" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-35" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-37" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-39" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-41" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-43" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-45" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-47" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-49" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-51" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-53" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-55" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-57" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-59" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-61" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-63" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-65" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-67" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-69" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-71" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-73" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-75" java.lang.NullPointerException
>>>>>>>> Exception in thread "Thread-87" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-89" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-91" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-93" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-95" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-97" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-99" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Exception in thread "Thread-101" java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.crosswire.jsword.book.ConcurrencyTest$2.run(ConcurrencyTest.java:57)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>>>>>>>> 0.123 sec
>>>>>>>>
>>>>>>>> This seems to be caused because
>>>>>>>> final Book b0 = Books.installed().getBook(names[1]);
>>>>>>>> returns empty.
>>>>>>>>
>>>>>>>> I do have modules installed - iterating over the
>>>>>>>> Books.installed().getBooks(); gives me a list of a half-dozen or so entries
>>>>>>>> when I execute it from within Eclipse, but none when I execute it from the
>>>>>>>> command line. I haven't had time to put my debugger on it yet to see what
>>>>>>>> the difference between the two environments is, though. They were installed
>>>>>>>> by the core Sword installmgr into my ~/.sword/ directory.
>>>>>>>>
>>>>>>>> --Greg
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 29, 2013 at 10:57 AM, DM Smith <dmsmith at crosswire.org>wrote:
>>>>>>>>
>>>>>>>>> I use ant to build and test. Maybe Chris B can chime in?
>>>>>>>>>
>>>>>>>>> I do know that there is a failure in some of the tests because
>>>>>>>>> under ant the order is deterministic and a prerequisite of later test is
>>>>>>>>> satisfied by an earlier one. Under Jenkins on Cloudbees, which is
>>>>>>>>> configured to use maven, it fails. At JavaOne, I talked to the CloudBees
>>>>>>>>> folks and we narrowed it down. But we didn't have time to solve it.
>>>>>>>>>
>>>>>>>>> Don't know if this is related to the failure that you are seeing.
>>>>>>>>>
>>>>>>>>> DM Smith
>>>>>>>>>
>>>>>>>>> On Oct 29, 2013, at 11:49 AM, Greg Hellings <
>>>>>>>>> greg.hellings at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> I was just trying to build JSword, but I can't run an 'mvn
>>>>>>>>> package' in the current github master because it fails with several NPEs
>>>>>>>>> during the Concurrency Test. Is this a known issue?
>>>>>>>>>
>>>>>>>>> Also, I can't locate a built jar of the core library in any maven
>>>>>>>>> repositories around. Is JSword not hosted in any of them?
>>>>>>>>>
>>>>>>>>> --Greg
>>>>>>>>> _______________________________________________
>>>>>>>>> jsword-devel mailing list
>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> jsword-devel mailing list
>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jsword-devel mailing list
>>>>>>>> jsword-devel at crosswire.org
>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jsword-devel mailing list
>>>>>>> jsword-devel at crosswire.org
>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> jsword-devel mailing list
>>>>>> jsword-devel at crosswire.org
>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> jsword-devel mailing list
>>>> jsword-devel at crosswire.org
>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> jsword-devel mailing list
>>>> jsword-devel at crosswire.org
>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>
>>>>
>>> _______________________________________________
>>> jsword-devel mailing list
>>> jsword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>
>>>
>>>
>>> _______________________________________________
>>> jsword-devel mailing list
>>> jsword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>
>>>
>> _______________________________________________
>> jsword-devel mailing list
>> jsword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20140210/7cf338a6/attachment-0001.html>
More information about the jsword-devel
mailing list