Gearman Java

Does java client gearman supports multithreading?

Asked by Raul Puente

I asked because i'm testing client gearman with N threads.

I've test 2 options.

1 - Using only one GearmanClientImpl and passing it to all the threads, but then i think that the client implemention don't recognize the jobs well. I see the next Warning:
WARNING: Client received packet from server for unknown job ( job_handle = org.gearman.client.GearmanClientImpl$JobHandle@3f65305d packet = WORK_COMPLETE )
org.gearman.common.GearmanException: Encountered Fatal Exception while driving the following ServerRequest: REQ:SUBMIT_JOB:1615: [66756c6c5f666565640036663836643736362d366430632d343230622d386139372d306330343538303338646639006172726179282769645f736f75726365273d3e2734272c2769645f6462273d3e273439272c27636f756e747279273d3e276573272c27766572746963616c273d3e2731272c276164273d3e617272617928617272617928276164273d3e6172726179282776616c7565273d3e27272c2761747472273d3e272729292c617272617928276964273d3e6172726179282776616c7565273d3e273539355f564134373036272c2761747472273d3e272729292c6172726179282775726c273d3e6172726179282776616c7565273d3e27687474703a2f2f7777772e757262616e697a612e636f6d2f6275736361722f696e6d7565626c65732f6669636861456e6c6163652e6a73703f635f6167656e6369613d35393526616d703b635f696e6d7565626c653d564134373036272c2761747472273d3e272729292c617272617928277469746c65273d3e6172726179282776616c7565273d3e274c6f63616c657320436f6d65726369616c657320656e20416c7175696c65722c203230206d322c204752414e414441204341504954414c20535552202856455247454c455329272c2761747472273d3e272729292c6172726179282774797065273d3e6172726179282776616c7565273d3e27466f722072656e74272c2761747472273d3e272729292c617272617928276167656e6379273d3e6172726179282776616c7565273d3e27557262616e697a612e636f6d272c2761747472273d3e272729292c61727261792827636f6e74656e74273d3e6172726179282776616c7565273d3e274c6f63616c657320436f6d65726369616c657320656e20416c7175696c65722c203230206d322c204752414e414441204341504954414c20535552202856455247454c4553292e20272c2761747472273d3e272729292c617272617928277072696365273d3e6172726179282776616c7565273d3e27323030272c2761747472273d3e272729292c6172726179282770726f70657274795f74797065273d3e6172726179282776616c7565273d3e274c6f63616c657320436f6d65726369616c6573272c2761747472273d3e272729292c61727261792827666c6f6f725f61726561273d3e6172726179282776616c7565273d3e273230272c2761747472273d3e61727261792827756e6974273d3e276d6574657273272929292c61727261792827726f6f6d73273d3e6172726179282776616c7565273d3e2730272c2761747472273d3e272729292c6172726179282762617468726f6f6d73273d3e6172726179282776616c7565273d3e2730272c2761747472273d3e272729292c617272617928277061726b696e67273d3e6172726179282776616c7565273d3e27272c2761747472273d3e272729292c6172726179282761646472657373273d3e6172726179282776616c7565273d3e27272c2761747472273d3e272729292c6172726179282763697479273d3e6172726179282776616c7565273d3e274752414e414441204341504954414c20535552272c2761747472273d3e272729292c61727261792827636974795f61726561273d3e6172726179282776616c7565273d3e2756455247454c4553272c2761747472273d3e272729292c61727261792827706f7374636f6465273d3e6172726179282776616c7565273d3e273138303038272c2761747472273d3e272729292c61727261792827726567696f6e273d3e6172726179282776616c7565273d3e274772616e616461272c2761747472273d3e272729292c617272617928276c61746974756465273d3e6172726179282776616c7565273d3e2733372e313831313038303837393135272c2761747472273d3e272729292c617272617928276c6f6e676974756465273d3e6172726179282776616c7565273d3e272d332e3630323331333939353336313332272c2761747472273d3e272729292c617272617928277069637475726573273d3e6172726179282776616c7565273d3e270a272c2761747472273d3e272729292c6172726179282764617465273d3e6172726179282776616c7565273d3e2732362f30392f32303039272c2761747472273d3e272729292c6172726179282774696d65273d3e6172726179282776616c7565273d3e2730323a3039272c2761747472273d3e272729292929]
        at org.gearman.common.GearmanTask.handleGearmanIOEvent(GearmanTask.java:114)
        at org.gearman.common.GearmanJobServerSession.handleResSessionEvent(GearmanJobServerSession.java:308)
        at org.gearman.common.GearmanJobServerSession.handleSessionEvent(GearmanJobServerSession.java:240)
        at org.gearman.common.GearmanJobServerSession.driveSessionIO(GearmanJobServerSession.java:203)
        at org.gearman.client.GearmanClientImpl.driveClientIO(GearmanClientImpl.java:547)
        at org.gearman.client.GearmanClientImpl.driveRequestTillState(GearmanClientImpl.java:599)
        at org.gearman.client.GearmanClientImpl.submit(GearmanClientImpl.java:269)
        at com.trovit.parsing.feeds.executers.ExternalExecuter.executeGearman(ExternalExecuter.java:124)

2- Using one GearmanClientImpl for earch Thread. Establishing connection. I test it with one thread and 10 threads.

The error is the same:
java.lang.InternalError
        at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:752)
        at sun.misc.URLClassPath.getResource(URLClassPath.java:168)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:192)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:300)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
        at java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:445)
        at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2383)
        at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1406)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1365)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:715)
        at java.util.logging.Level.getLocalizedName(Level.java:223)
        at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:64)
        at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
        at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
        at java.util.logging.Logger.log(Logger.java:472)
        at java.util.logging.Logger.doLog(Logger.java:494)
        at java.util.logging.Logger.log(Logger.java:583)
        at org.gearman.client.GearmanClientImpl.addJobServer(GearmanClientImpl.java:168)
        at com.trovit.parsing.feeds.executers.ExternalExecuter.executeGearman(ExternalExecuter.java:106)
        at com.trovit.parsing.feeds.executers.FullFeedExecuter.run(FullFeedExecuter.java:147)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
        at java.util.jar.JarFile.<init>(JarFile.java:133)
        at java.util.jar.JarFile.<init>(JarFile.java:70)
        at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:643)
        at sun.misc.URLClassPath$JarLoader.access$600(URLClassPath.java:538)
        at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:605)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:597)
        at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:750)
        ... 29 more

It seems problem with Log from java:
            LOG.log(Level.WARNING,"Failed to connect to job server " + newconn + ".",ioe);
i change it to test -> System.out.println("Failed to connect to job server " + newconn + ".") ;

And then i receive the error:
Failed to connect to job server GearmanNIOJobServerConnection:/10.0.0.106:4730.
java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.initPipe(Native Method)
        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
        at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
        at java.nio.channels.Selector.open(Selector.java:209)
        at org.gearman.client.GearmanClientImpl.addJobServer(GearmanClientImpl.java:161)
        at com.trovit.parsing.feeds.executers.ExternalExecuter.executeGearman(ExternalExecuter.java:106)
        at com.trovit.parsing.feeds.executers.FullFeedExecuter.run(FullFeedExecuter.java:147)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

The code i use from this in the thread is:
GearmanClientImpl gearman = new GearmanClientImpl();
gearman.addJobServer(new GearmanNIOJobServerConnection(hostname, port));
GearmanJob job = GearmanJobImpl.createJob(this.gearmanFunction, org.gearman.util.ByteUtils.toUTF8Bytes(message) , null);
gearman.submit(job);
...
gearman.shutdown() ;

I'm doing something wrong?

Thanks,
Raul

Question information

Language:
English Edit question
Status:
Answered
For:
Gearman Java Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Raul Puente (raulpuente) said : 		#1

I think i've seen the error.

The real problem is: java.io.IOException: Too many open files.

I was inspecting with lsof command the process i was running and i saw that the pipes was increasing and they were never closed. Viewing the code i've added some lines in class: GearmanJobServerSession.java.

Exactly i've added the variable

- private Selector sel = null ;

- In function initSession i reference the variable: this.sel = sel ;

- In function closeSession i have added:
        try {
            sel.close();
        } catch (IOException ex) {
            LOG.log(Level.SEVERE,"Error closing selector", ex);
        }

And pipes haven't increased more.

If you want i can send to you the java class.

Thanx.

Revision history for this message
Launchpad Janitor (janitor) said : 		#2

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Eric Lambert (elambert) said : 		#3

Hi Raul:

Wow, very sorry about taking 5 months to reply to this question, but I somehow managed to miss this question being posted. It was not until one of your colleagues, Norberto, filed a bug and mentioned your original question did I realize that I had missed it. Please accept my very sincere apologies. I am rather embarrassed that it went unnoticed for so long.

Your fix looks reasonable to me, I'll see what I can do about getting it into the trunk as soon as possible.

Eric

Revision history for this message
Raul Puente (raulpuente) said : 		#4

Thank you very much Eric :)

On Fri, Sep 10, 2010 at 10:20 PM, Eric Lambert <
<email address hidden>> wrote:

> Bug #513180 status changed in Gearman Java:
>
> Fix Committed => Fix Released
>
>
> https://bugs.launchpad.net/gearman-java/+bug/513180
> "Selector not being closed"
>
> This bug is linked to #83913.
> Does java client gearman supports multithreading?
> https://answers.launchpad.net/gearman-java/+question/83913
>
> --
> You received this question notification because you are a direct
> subscriber of the question.
>

Revision history for this message
Eric Lambert (elambert) said : 		#5

Raul ... sure thing ... sorry it took so long :-)

Can you help with this problem?

Provide an answer of your own, or ask Raul Puente for more information if necessary.

To post a message you must log in.