Thursday, December 19, 2013

Lambda'ery WebSocket code (from UKTECH13 presentation)

At UKTECH13 I was asked to post the source code to the WebSocket used in my presentation, primarily because it was using JDK 8 constructs that were unfamiliar to many. One of the very nice things about the changes to the languages and the supporting library changes is the lack of if statements in the code.

I would note that in a real world chat application that it unlikely different rooms would have different resource URIs but for the purposes of the presentation this made sense.

package websocket;

import static java.util.Collections.emptySet;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CopyOnWriteArraySet;

import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.PathParam;
import javax.websocket.server.ServerEndpoint;

@ServerEndpoint(value = "/chat/{room}")
public class ChatService {

  private static final Set<Session> EMPTY_ROOM = emptySet();

  static final ConcurrentMap<String, Set<Session>> rooms =
    new ConcurrentHashMap<>(); 


  @OnOpen
  public void onOpen(Session peer, @PathParam("room") String room) {
    rooms.computeIfAbsent(room,
                          s -> new CopyOnWriteArraySet<Session>()).add(peer);
  }

  @OnClose
  public void onClose(Session peer, @PathParam("room") String room) {

    rooms.getOrDefault(room, EMPTY_ROOM).remove(peer);
    
  }

  @OnError
  public void onError(Session peer, Throwable th,
                      @PathParam("room") String room) {
    System.out.println("Peer error " + room + " " + th);
  }


  @OnMessage
  public void message(String message, Session peer,
                      @PathParam("room") String room) {

    // Send a message to all peers in a room who are not the current
    // peer and are still open. Send the message asynchronously to ensure
    // that the first client is not hung up. 

    rooms.getOrDefault(room, EMPTY_ROOM).parallelStream()
         .filter(s -> s != peer && s.isOpen())
         .forEach(s -> s.getAsyncRemote().sendObject(message));
  };

}

One of the problem with the above design is that there is no error logging when an invalid room is used. This could be useful to diagnose errors. Not wanting to use any conditional statements you could use an Optional object:

import java.util.Optional;


  @OnClose
  public void onClose(Session peer, @PathParam("room") String room) {

    Optional.ofNullable(rooms.get(room))
      .orElseThrow(() -> new IllegalStateException("Cannot find room " + room))
      .remove(peer);
    
  }


Of course you might want this on your method objects, so you can use default methods to create a mixin for this on all your Map objects with a trivial subclass.

import java.util.Optional;


  public interface OptionalMap<K,V> extends ConcurrentMap<K,V> {
     public default Optional<V> find(K key) {
        return Optional.ofNullable(get(key));
     }
  }

  public static class OptionalHashMap<K,V> extends ConcurrentHashMap<K,V>
   implements OptionalMap<K,V> {
       
  }
    

  static final OptionalMap<String, Set<Session>> rooms =
    new OptionalHashMap<>(); 


  @OnClose
  public void onClose(Session peer, @PathParam("room") String room) {

    rooms.find(room)
      .orElseThrow(() -> new IllegalStateException("Cannot find room " + room))
      .remove(peer);
    
  }





Whilst working on my presentation it became apparent that it was also possible to use the "openSessions" and "getUserProperties" method to store discrimination data against the Session. I don't have enough experience yet to say which is the better design for a particular case.

package websocket;


import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.PathParam;
import javax.websocket.server.ServerEndpoint;

@ServerEndpoint(value = "/chat/{room}")
public class ChatService {

  private static final String ROOM_PROPERTY = "ROOM";


  @OnOpen
  public void onOpen(Session peer, @PathParam("room") String room) {
    peer.getUserProperties().put(ROOM_PROPERTY, room);
  }

  @OnClose
  public void onClose(Session peer, @PathParam("room") String room) {
    
    // No need to tidy up and data is store against peer
  }

  @OnError
  public void onError(Session peer, Throwable th,
                      @PathParam("room") String room) {
    System.out.println("Peer error " + room + " " + th);
  }


  @OnMessage
  public void message(String message, Session peer,
                      @PathParam("room") String room) {

    peer.getOpenSessions().parallelStream()
         .filter(s -> room.equals(s.getUserProperties().get(ROOM_PROPERTY)))
         .filter(s -> s != peer && s.isOpen())
         .forEach(s -> s.getAsyncRemote().sendObject(message));
  };

}

Wednesday, December 4, 2013

Lambda, will it serialize?

So I have been ponder an enhancement required on the Tyrus project that would allow a user to broadcast to a subset of client connected to a URL across a cluster of machines. There are various way of doing this; but since I was playing with JDK 8 this problem definitely looked like a nail.

To this end I created a simple unit test class that would take my filter, serialise it to disk, read it back and in then execute it. It had a instance field "VALUE" that we could use to reference directly or indirectly to find out what would cause the serialisation to fail.

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.NotSerializableException;
import java.io.ObjectInput;
import java.io.ObjectInputStream;
import java.io.ObjectOutput;
import java.io.ObjectOutputStream;
import java.io.Serializable;

import java.util.function.Predicate;

import org.junit.Test;

public class SerializablePredicateFilterTest {

  public String VALUE = "Bob";

  public interface SerializablePredicate<T> extends Predicate<T>, Serializable {

  }


  public <T> void filter(SerializablePredicate<T> sp, T value) throws IOException, ClassNotFoundException {


    sp.getClass().isLocalClass();

    File tempFile = File.createTempFile("labmda", "set");


    try (ObjectOutput oo = new ObjectOutputStream(new FileOutputStream(tempFile))) {
      oo.writeObject(sp);
    }


    try (ObjectInput oi = new ObjectInputStream(new FileInputStream(tempFile))) {
      SerializablePredicate<T> p = (SerializablePredicate<T>) oi.readObject();

      System.out.println(p.test(value));
    }


  }

}


So just to calibrate lets make sure that an anonymous inner class will fail, because it will always contain a reference to enclosing object....

  @Test(expected = NotSerializableException.class)
  public void testAnonymousDirect() throws IOException, ClassNotFoundException {

    String value = VALUE;


    filter(new SerializablePredicate<String>() {

      @Override
      public boolean test(String t) {
        return value.length() > t.length();
      }
    }, "Bob");

  }


The same is true for local classes, what you don't use local classes?

  @Test(expected = NotSerializableException.class)
  public void testLocalClass() throws IOException, ClassNotFoundException {

    class LocalPredicate implements SerializablePredicate<String> {
      @Override
      public boolean test(String t) {
        // TODO Implement this method
        return false;
      }
    }

    filter(new LocalPredicate(), "Bobby");

  }


So a standalone class will of course work, in this case a nested class for convenience.

  public static class LengthPredicate implements SerializablePredicate<String> {

    private String value;


    public LengthPredicate(String value) {
      super();
      this.value = value;
    }

    public void setValue(String value) {
      this.value = value;
    }

    public String getValue() {
      return value;
    }

    @Override
    public boolean test(String t) {
      // TODO Implement this method
      return false;
    }
  }


  @Test
  public void testStaticInnerClass() throws IOException, ClassNotFoundException {

    filter(new LengthPredicate(VALUE), "Bobby");

  }


So lets get down with JDK 8, it turns out that my first try also fails but it does confirm that the serialisation is quite happy to take a Lambda in general.

  @Test(expected = NotSerializableException.class)
  public void testLambdaDirect() throws IOException, ClassNotFoundException {


    filter((String s) -> VALUE.length() > s.length(), "Bobby");

  }


A slight modification to copy the value into a effectively final attributes, and voila the lambda is now serialised and retrieved properly.

  @Test
  public void testLambdaInDirect() throws IOException, ClassNotFoundException {

    String value = VALUE;

    filter((String s) -> value.length() > s.length(), "Bobby");

  }


And of course if the value is a simple method parameter it also works fine.


  @Test
  public void testLambdaParameter() throws IOException, ClassNotFoundException {

    invokeWithParameter(VALUE);

  }

  private void invokeWithParameter(String value) throws java.lang.ClassNotFoundException, java.io.IOException {
    filter((String s) -> value.length() > s.length(), "Bobby");
  }


So the answer is yes, you can get it to serialise if you are a bit careful.

Wednesday, October 2, 2013

Write an auto-debugger to catch Exceptions during test execution

Previously I have stated that there are some exceptions you would always want to keep an debugger breakpoint on for. This help prevents code rotting away without you noticing - sometimes masking a different problem.

If you take this seriously then it is a good idea to extend this idea to you automated testing; but coming up with a comprehensive solution is not entirely trivial. You could just start with a try/catch but that won't capture exceptions on other threads. You could also do something using AOP; but depending on the framework you are not guaranteed to catch everything also it does mean you are testing with slightly different code which will worry some.

A few days ago I came across this blog entry on how to write your own debugger, and I wondered if it was possible for java process to debug itself. Turn out you can and here is the code I came up with as part of this little thought experiment.

The first part of the class just contains something fairly hacky code to guess what the port that would be required to connect back to the same VM based on the start up parameters. It might be possible to use the Attach mechanism to start the debugger; but I didn't seen an obvious way to get it to work. Then there are just a couple of factory methods that take a list of exceptions to look out for.

package com.kingsfleet.debug;

import com.sun.jdi.Bootstrap;
import com.sun.jdi.ReferenceType;
import com.sun.jdi.VirtualMachine;
import com.sun.jdi.connect.AttachingConnector;
import com.sun.jdi.connect.Connector;
import com.sun.jdi.connect.IllegalConnectorArgumentsException;
import com.sun.jdi.event.ClassPrepareEvent;
import com.sun.jdi.event.Event;
import com.sun.jdi.event.EventQueue;
import com.sun.jdi.event.EventSet;
import com.sun.jdi.event.ExceptionEvent;
import com.sun.jdi.event.VMDeathEvent;
import com.sun.jdi.event.VMDisconnectEvent;
import com.sun.jdi.request.ClassPrepareRequest;
import com.sun.jdi.request.EventRequest;
import com.sun.jdi.request.ExceptionRequest;

import java.io.IOException;

import java.lang.management.ManagementFactory;
import java.lang.management.RuntimeMXBean;

import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;


public class ExceptionDebugger implements AutoCloseable {




   public static int getDebuggerPort() {
       // Try to work out what port we need to connect to

       RuntimeMXBean runtime = ManagementFactory.getRuntimeMXBean();
       List<String> inputArguments = runtime.getInputArguments();
       int port = -1;
       boolean isjdwp = false;

       for (String next : inputArguments) {
           if (next.startsWith("-agentlib:jdwp=")) {
               isjdwp = true;
               String parameterString = next.substring("-agentlib:jdwp=".length());
               String[] parameters = parameterString.split(",");
               for (String parameter : parameters) {
                   if (parameter.startsWith("address")) {
                       int portDelimeter = parameter.lastIndexOf(":");
                       if (portDelimeter != -1) {
                           port = Integer.parseInt(parameter.substring(portDelimeter + 1));
                       } else {
                           port = Integer.parseInt(parameter.split("=")[1]);
                       }
                   }
               }
           }
       }
       return port;
   }


   public static ExceptionDebugger connect(final String... exceptions) throws InterruptedException {
       return connect(getDebuggerPort(),exceptions);
   }

   public static ExceptionDebugger connect(final int port, final String... exceptions) throws InterruptedException {

       ExceptionDebugger ed = new ExceptionDebugger(port, exceptions);

       return ed;
   }


The constructor create a simple daemon thread that starts the connection back to the VM. It is very important to do this is a separate thread otherwise obviously the VM will grind to a halt when we hit a breakpoint. It is a good idea to make sure that code in that thread doesn't throw one of the exceptions - for the moment I am just hoping for the best.

Finally the code just maintains a list of the banned exceptions, if you had a bit more time it should be possible to store the stack trace where the exception occurred.

// 
   // Instance variables

   private final CountDownLatch startupLatch = new CountDownLatch(1);
   private final CountDownLatch shutdownLatch = new CountDownLatch(1);

   private final Set<String> set = Collections.synchronizedSet(new HashSet<String>());
   private final int port;
   private final String exceptions[];
   private Thread debugger;
   private volatile boolean shutdown = false;


   //
   // Object construction and methods
   //




   private ExceptionDebugger(final int port, final String... exceptions) throws InterruptedException {

       this.port = port;
       this.exceptions = exceptions;

       debugger = new Thread(new Runnable() {

           @Override
           public void run() {
               try {
                   connect();
               } catch (Exception ex) {
                   ex.printStackTrace();
               }
           }
       }, "Self debugging");
       debugger.setDaemon(true); // Don't hold the VM open
       debugger.start();

       // Make sure the debugger has connected
       if (!startupLatch.await(1, TimeUnit.MINUTES)) {
           throw new IllegalStateException("Didn't connect before timeout");
       }
   }


   @Override
   public void close() throws InterruptedException {
       shutdown = true;
       // Somewhere in JDI the interrupt was being eaten, hence the volatile flag 
       debugger.interrupt();
       shutdownLatch.await();
   }


   /**
    * @return A list of exceptions that were thrown
    */
   public Set<String> getExceptionsViolated() {
       return new HashSet<String>(set);
   }

   /**
    * Clear the list of exceptions violated
    */
   public void clearExceptionsViolated() {
       set.clear();
   }


The main connect method is a fair simple block of code that ensures the connection and configures any initial breakpoints.

//
   // Implementation details
   //


   private void connect() throws java.io.IOException {


       try {
           // Create a virtual machine connection
           VirtualMachine attach = connectToVM();


           try
           {

               // Add prepare and any already loaded exception breakpoints
               createInitialBreakpoints(attach);

               // We can now allow the rest of the work to go on as we have created the breakpoints
               // we required

               startupLatch.countDown();

               // Process the events
               processEvents(attach);
           }
           finally {

               // Disconnect the debugger
               attach.dispose();

               // Give the debugger time to really disconnect
               // before we might reconnect, couldn't find another
               // way to do this

               try {
                   TimeUnit.SECONDS.sleep(1);
               } catch (InterruptedException e) {
                   Thread.currentThread().interrupt();
               }
           }
       } finally {
           // Notify watchers that we have shutdown
           shutdownLatch.countDown();
       }
   }



Connecting back to self is just a process of finding the right attaching connector, in this case Socket although I guess you could use the shared memory transport on some platforms if you modified the code slightly.

private VirtualMachine connectToVM() throws java.io.IOException {

       List<AttachingConnector> attachingConnectors = Bootstrap.virtualMachineManager().attachingConnectors();
       AttachingConnector ac = null;

       found:
       for (AttachingConnector next : attachingConnectors) {
           if (next.name().contains("SocketAttach")) {
               ac = next;
               break;

           }
       }

       Map<String, Connector.Argument> arguments = ac.defaultArguments();
       arguments.get("hostname").setValue("localhost");
       arguments.get("port").setValue(Integer.toString(port));
       arguments.get("timeout").setValue("4000");

       try {
           return ac.attach(arguments);
       } catch (IllegalConnectorArgumentsException e) {
           throw new IOException("Problem connecting to debugger",e);
       }
   }


When you connect the debugger you have no idea as to whether the exceptions you are interested in have been loaded, so you need to register breakpoints for both the point where the classes are prepared and for those that have already been loaded.

Note that the breakpoint is set with a policy only to break the one thread - otherwise for obvious reasons the current VM will grind to a halt if the debugger thread is also put to sleep.

private void createInitialBreakpoints(VirtualMachine attach) {
       // Our first exception is for class loading

       for (String exception : exceptions) {
           ClassPrepareRequest cpr = attach.eventRequestManager().createClassPrepareRequest();
           cpr.addClassFilter(exception);
           cpr.setSuspendPolicy(EventRequest.SUSPEND_EVENT_THREAD);
           cpr.setEnabled(true);
       }

       // Then we can check each in turn to see if it have already been loaded as we might
       // be late to the game, remember classes can be loaded more than once
       //

       for (String exception : exceptions) {
           List<ReferenceType> types = attach.classesByName(exception);
           for (ReferenceType type : types) {
               createExceptionRequest(attach, type);
           }
       }
   }


   private static void createExceptionRequest(VirtualMachine attach, 
                                              ReferenceType refType) {
       ExceptionRequest er = attach.eventRequestManager().createExceptionRequest(
           refType, true, true);
       er.setSuspendPolicy(EventRequest.SUSPEND_EVENT_THREAD);
       er.setEnabled(true);
   }



The event processing loop polls for EventSet instances which contain one or more Event instances. Not all of these events are down to a breakpoint request though so you have to take care to not always call resume on the event set. This is because you might have two event sets in a row, with the code calling resume before you even get to read the second one. This results in missed breakpoints as the code caught up.

For some reason JDI appeared to be eating the interrupted flag, hence the boolean property to stop the loop with the close method from before.

private void processEvents(VirtualMachine attach) {
       // Listen for events

       EventQueue eq = attach.eventQueue();
       eventLoop: while (!Thread.interrupted() && !shutdown) {

           // Poll for event sets, with a short timeout so that we can
           // be interrupted if required
           EventSet eventSet = null;
           try 
           {
               eventSet = eq.remove(500);
           }
           catch (InterruptedException ex) {
               Thread.currentThread().interrupt();
               continue eventLoop;  
           }

           // Just loop again if we have no events
           if (eventSet == null) {
               continue eventLoop;
           }

           //

           boolean resume = false;
           for (Event event : eventSet) {

               EventRequest request = event.request();
               if (request != null) {
                   int eventPolicy = request.suspendPolicy();
                   resume |= eventPolicy != EventRequest.SUSPEND_NONE;
               }

               if (event instanceof VMDeathEvent || event instanceof VMDisconnectEvent) {
                   // This should never happen as the VM will exit before this is called

               } else if (event instanceof ClassPrepareEvent) {

                   // When an instance of the exception class is loaded attach an exception breakpoint
                   ClassPrepareEvent cpe = (ClassPrepareEvent) event;
                   ReferenceType refType = cpe.referenceType();
                   createExceptionRequest(attach, refType);

               } else if (event instanceof ExceptionEvent) {

                   String name = ((ExceptionRequest)event.request()).exception().name();
                   set.add(name);
               }
           }

           // Dangerous to call resume always because not all event suspend the VM
           // and events happen asynchornously.
           if (resume)
               eventSet.resume();
       }
   }

}

So all that remains is a simple test example, since this is JDK 7 and the ExceptionDebugger is AutoCloseable we can do this using the try-with-resources construct as follows. Obviously if doing automated test use the testing framework fixtures of your choice.

public class Target {

   public static void main(String[] args) throws InterruptedException {


       try (ExceptionDebugger ex = ExceptionDebugger.connect(
               NoClassDefFoundError.class.getName())) {

           doSomeWorkThatQuietlyThrowsAnException();

           System.out.println(ex.getExceptionsViolated());
       }


       System.exit(0);
   }


   private static void doSomeWorkThatQuietlyThrowsAnException() {
       // Check to see that break point gets fired

       try {
           Thread t = new Thread(new Runnable() {
                           public void run() {
                               try
                               {
                                   throw new NoClassDefFoundError();
                               }
                               catch (Throwable ex) {

                               }
                           }
                      });
           t.start();
           t.join();
       } catch (Throwable th) {
           // Eat this and don't tell anybody
       }
   }
}


So if you run this class with the following VM parameter, note the suspend=n otherwise the code won't start running, you will find that it can connect back to itself and start running.

-agentlib:jdwp=transport=dt_socket,address=localhost:5656,server=y,suspend=n

This gives you the following output, note the extra debug line from the VM:

Listening for transport dt_socket at address: 5656
[java.lang.NoClassDefFoundError]
Listening for transport dt_socket at address: 5656

As every I would be interested to read if this was something that is useful for people and to help remove any obvious mistakes.

Friday, August 9, 2013

Almost named method arguments in JDK 8

Sometimes it would be really nice to have named method parameter in Java, this doesn't look likely for a long while but there is always another little work around, for example using a builder pattern to get similar behaviour, that will bridge the gap for a little bit. It occurred to me that using the Lambda support in JDK 8 you could get something very close without the boiler plate required by some of the other methods.

So consider this simple Facade class, that presents a method with a variable list of parameters. Note the variant of the method that takes in a Consumer object - this is what we are going to invoke with a Lambda later on.

package client;

import java.util.function.Consumer;

public class Facade {
    
    public static void invoke(Consumer<Parameter> op) {
        
        Parameter p = new Parameter();
        op.accept(new Parameter());
        invoke(p);        
    }
    
    public static void invoke(Parameter p) {

        // Defensive copy and validate
        p = p.clone();
        p.validate();
        
        // ...
    }

    
    public static class Parameter implements Cloneable {
        public String value1, value2, value3;
        
        public Parameter validate() {
            // ...
            
            return this;
        }

        public Parameter clone() {
            // We can use vanilla clone here, for more complicated mutable objects
            // you might need to perform a deeper copy
            try {
                return (Parameter) super.clone();
            } catch (CloneNotSupportedException e) {
                throw new RuntimeException(e);
            }
        }
    }
    
}

So once you have this in place, you can use Lambda notation to call the invoke method with how every many parameters as you might think suitable:

...
{

    public static void main(String[] args) {
        
        // Single parameter version
        Facade.invoke(p ->  p.value1="x" );
        // Multiple parameter version
        Facade.invoke(p -> { p.value1="x"; p.value2="y";});
    }
}

Of course the eagle eyed of your will recognise that the same pattern, minus the Consumer method of course, with an earlier JDK. Given the number of java developers out there some-one probably will have probably already invented this pattern; but you never know and if you don't write it down the thought it didn't happen.

...
{

    public static void main(String[] args) {
        
        // Using an anonymous inner class pre JDK 8
        Facade.invoke(new Parameter() {{
            value1 = "x";
            value2 = "y";
        }});
    }
}

Thursday, June 6, 2013

Problems running lots of X11 sessions concurrently when testing under Hudson/Jenkins

So we are running a quite a bit of automated UI testing and we have found over a large number of concurrent test nodes that even when the Xvnc plugin has correctly started that the environment is not always ready or reliable. This blog post looks at two changes we made that have improved our test stability in recent times.

The first and most important thing to settle when using the Xvnc plugin is to make sure that the user that is running the tests has a normal window manager. By default the default VNC server will start using "twm" as defined in the .vnc/xstartup file. This has significant behavioural differences from the more standard Gnome / KDE and indeed the window managers on Windows and a Mac. Rather than deal with this we have standardised on running with Gnome.

The problem is though is that in our experience if you have multiple gnome sessions starting at the same time that sometimes they will fail with odd error messages. My esteemed colleague Mark Warner came up with the following xstartup script which just puts a simple lock around the startup code.

#!/bin/bash
# It uses a lockfile in the user's .vnc directory to ensure that
# only one instance of gnome will start up for the given user
# at once. This should work around problems associated with
# multiple gnome instances starting and failing to lock certain
# files. It will start gnome after 5 mins of failing to lock as
# a fail safe so a gnome startup will always be attempted.
#

LOCK_FILE=${HOME}/.vnc/xstartup.lock
SESSION_VNC_NAME=VNC-`hostname -f`
LOCKED=0

function clean_up {
  # don't delete the lock file if we didn't create it
  if [ $LOCKED -eq 1 ]; then
    echo "$(date) All finished, deleting lock file."
    rm -f $LOCK_FILE
  fi
}

# make sure that we delete the lock file if we get killed
trap clean_up SIGHUP SIGINT SIGTERM

# retry for 5 mins then assume something died and delete
# the existing lockfile taking control of it for ourselves.
echo "$(date) Attempting to obtain startup lock."
if lockfile -l 300 $LOCK_FILE; then
  echo "$(date) Startup lock aquired."
  LOCKED=1
else
  echo "$(date) Lock failed... starting anyway."
fi

# give X a little breather before we start gnome
sleep 10 

# Start Gnome
gnome-session &

# just let gnome settle before we relinquish the lock
sleep 20
clean_up

Now this resolved spurious test failures that were due to a missing window manager, but as recently the number of nodes increased we found that we were still seeing troubling intermittent failures that took the form of failures to connect to the window server for no obvious reason.

This would happen even if previously in the job we had displayed a window successfully.

[exec] java.lang.InternalError: Can't connect to X11 window server using ':58' as the value of the DISPLAY variable.
     [exec]  at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
     [exec]  at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
     [exec]  at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:110)
     [exec]  at java.security.AccessController.doPrivileged(Native Method)
     [exec]  at sun.awt.X11GraphicsEnvironment.(X11GraphicsEnvironment.java:74)
     [exec]  at java.lang.Class.forName0(Native Method)
     [exec]  at java.lang.Class.forName(Class.java:186)
     [exec]  at java.awt.GraphicsEnvironment.createGE(GraphicsEnvironment.java:102)
     [exec]  at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:81)
     [exec]  at org.netbeans.core.startup.Main.start(Main.java:253)
     [exec]  at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:123)
     [exec]  at java.lang.Thread.run(Thread.java:722)
     [exec] Result: 2

This would also happen right at the start when gnome was being set up, as shown here by looking at the content of the logs under .vnc .

_XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root

Xvnc Free Edition 4.1.2
Copyright (C) 2002-2005 RealVNC Ltd.
See http://www.realvnc.com for information on VNC.
Underlying X server release 70101000, The X.Org Foundation


Thu May 30 03:51:34 2013
 vncext:      VNC extension running!
 vncext:      Listening for VNC connections on port 5974
 vncext:      Listening for HTTP connections on port 5874
 vncext:      created VNC server for screen 0
Thu May 30 03:51:37 BST 2013 Attempting to obtain startup lock.
Thu May 30 03:53:45 BST 2013 Startup lock aquired.
Start gnome for OL4 or OL5
AUDIT: Thu May 30 03:53:56 2013: 10280 Xvnc: client 1 rejected from local host
Xlib: connection to ":74.0" refused by server
Xlib: No protocol specified


(gnome-session:10543): Gtk-WARNING **: cannot open display:  
Thu May 30 03:54:15 BST 2013 All finished, deleting lock file.
AUDIT: Thu May 30 04:14:03 2013: 10280 Xvnc: client 1 rejected from local host
AUDIT: Thu May 30 04:14:22 2013: 10280 Xvnc: client 1 rejected from local host
FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing.

After a little bit of playing around suspicion fell to the .Xauthority file we we knew could cause error messages like this and could become corrupted. Removing /zeroing the file didn't help so like the Gnome problems some kind of concurrency / contention issues was suspected.

After a bit of playing about we decided that for our test user it was going to be best to just have a copy of this file per host. This works for us as we only have the single executor per machine in order to get consistent testing results. In our case we just added the following line in our .bashrc for the NIS user we run our tests as and restarted all our of slave nodes.

# make sure we don't share the Xauthority file between hosts

export XAUTHORITY=~/.Xauthority.`hostname -f`

Since we have made this change we have seen no failures of the above type after nearly week, that of course doesn't rule out that we have just reduced the occurrence of the issue to a reasonably small level; but we are happy that we can run tests again. Of course the moment I post this blog I am going to loose that lovely run of blue balls. :-)



Friday, May 24, 2013

wadl 1.1.5 released

I should have done this a while ago but last month we released a new version of the wadl2java tools, primarily to deal with issues seen whilst writing a previous blog. The headline change was a move to the latest version of the jsonschema2pojo in order to resolve some class generation issues. Otherwise if you are just using XML Schema based WADL there is nothing new in this release.

Thursday, May 23, 2013

Persistent breakpoints every java developer should have

When a developer is working in Java there are a few failure cases you always want to know about even if you were trying to debug something else. Here is a list of the persistent breakpoints that should be enabled in every IDE. (IMHO of course)

Yes in theory you should be able to just get this from a good logging implementation; but often these problems are deep down in somebody else's library caused by changes that are beyond your control..... or just hidden in a large number of currently executing threads.

Java these days doesn't have that much of an overhead to be running in debug mode, generally I never run anything I am working on without a debugger attached as HotSwap is such a productivity boost.

I would be interested to heard of other people's must have breakpoints. They should be failures that should never come up in the normal running of your application that you would want to know about right away.

Deadlock detection


The VM is getting better and better at recognising deadlocks from low level java contracts, you debugger should have an option to break on deadlock. You should make sure it is turned on by default.

On java.lang.ExceptionInInitializer


This one can be a right pain to track down because further down the line it can become a ClassNotFoundException and often if it is more than a few levels down there is no obvious reason why the indirectly referencing class has failed to load.

Often caused by people initialising public/final/static variables with methods/constructors that throw RuntimeExceptions. If you are not sure use a static{} block with proper error handling, never do something in there that depends on something external to the current jar file.

On java.lang.NoClassDefFoundError


Always a popular when working in OSGi, normally you have forgotten a dependant jar file and you see this type of failure. Interestingly you sometimes only get a cause for the first time this exception is thrown for a particular class. So if you have a breakpoint then you can more quickly track this down.

I have found that sometimes that the breakpoint is not hit when behind some reflective code. In that case I often have breakpoints in the constructor of this class just to make sure.

On java.lang.NoSuchMethodError


You will of course generally see this if you haven't built locally properly or you are using miss-matched versions of libraries. You always want to know about this as soon as it happens.

On java.lang.LinkageError


This normally only turns up when things really go bad, for example when you have two versions of the same class hanging about even though they are from the same jar file. (Gotta love multiple class loaders) Be thankful these don't turn up very often, normally timed for a day when I have a really bad headache already.

On java.lang.OutOfMemoryError / java.lang.StackOverflowException


If you are very lucky the VM will breakpoint for these; but unfortunately as a lot of time this will happen in native code you will just have to deal with the fallout.

Getting the full stack for the latter is a bit of a pain; but not impossible. See this older post on getting the full stack trace.

On java.lang.AssertionError


A surprising number of people use assertions and then get upset when you complain about there exceptions when debugging or running tests. (You are doing this with -ea aren't you?)

The downside is that you want to turn this off when debugging certain testing frameworks, I am looking at you JUnit, as it is not entirely helpful to hit a breakpoint when you just want to run all of your tests.

Finally every now and again I will run across the following code, which causes an exception just to test if assertions are enabled.

boolean assertionsEnabled = false;
    try
    { assert false; }
    catch (AssertionException ae)
    { assertionsEnabled = true }

Update: It has been pointed out by a helpful colleague at that I should have provided some alternative suggestions as to what this code should be replaced with. The first option would be to call Class.desiredAssertionStatus but if you want to be totally sure you can use the following code that uses assignment to capture the assertion status without throwing an exception.

boolean assertionsEnabled = false;

    assert isAssertionsEnabled = true;

Wednesday, May 22, 2013

A little command line Web Socket client tool from Tyrus

The Tyrus project, the RI implementation of the Web Socket JSR, has a little sub project added by Pavel Bucek that provides a very useful command line testing tool. (Along with a couple of bug fixes of my own) This blog post gives a quick introduction.

To get hold of the tool at the moment you are going to have to do a little bit of work with svn and maven. So you need to check out the head of https://svn.java.net/svn/tyrus~source-code-repository/trunk and then perform a mvn package on the etc/client-cli sub project. Depending on the state of the maven repository you might need to do a mvn install -DskipTests=true on the root project first.

You can now find the executable jar in .../tyrus/ext/client-cli/target called something like tyrus-client-cli....jar. This has been shaded so it contains all of the required deps to run. Simples.

The tool currently allows you to send text and ping messages. It can receive binary messages but it just displays the hex values of up to the first 1024 bytes.

So you can either pass in the ws:// URL as the single command line argument or you can connect using the open command once the tool is running. Here are some very simple commands along with a response from a simple echo web socket service.

gdavison@gbr10460 ~]$ java -jar ~/software/tyrus-client-cli-1.0-SNAPSHOT.jar ws://localhost:7101/Project1/echo
# Connecting to ws://localhost:7101/Project1/echo...
# Connected in session 689607b4-7bd8-465a-8d84-e0dca54fa3e8
session 6896...a3e8> 
session 6896...a3e8> send Hello!
# text-message: Message out Hello!
session 6896...a3e8> send
# End multiline message with . on own line
send...> Some
send...> Long
send...> Message
send...> .
# text-message: Message out Some
# Long
# Message
# 
session 6896...a3e8> close
# closed: CloseReason[1000,no reason given]
# Session closed
tyrus-client> quit


gdavison@gbr10460 ~]$ java -jar ~/software/tyrus-client-cli-1.0-SNAPSHOT.jar 

tyrus-client> 
tyrus-client> open ws://localhost:7101/Project1/echo
# Connecting to ws://localhost:7101/Project1/echo...
# Connected in session 59e501a9-a793-4321-ba6a-1c73c13bd822
session 59e5...d822>
session 59e5...d822> ping
# pong-message
session 59e5...d822> help
# 
    open uri : open a connection to the web socket uri
    close : close a currently open web socket session
    send message : send a text message
    send : send a multiline text message teminated with a .
    ping : send a ping message
    quit | exit : exit this tool
    help : display this message
    
session 59e5...d822> 

I can see this being very helpful when trying to debug a web socket service and you want to remove the web browser out of the equation. It is implemented using JLine2 so it behaves like a native command line tool so is pleasant to use.

Tuesday, April 30, 2013

Using Java WebSockets, JSR 356, and JSON mapped to POJO's

So I have been playing around with Tyrus, the reference implementation of the JSR 356 WebSocket for Java spec. Because I was looking at test tooling I was interested in running both the client and the server side in Java. So no HTML5 in this blog post I am afraid.

In this example we want to sent JSON back and forth and because I am old fashioned like that I want to be able to bind to a POJO object. I am going to use Jackson for this so my maven file looks like this:

<dependencies>
    <dependency>
        <groupId>javax.websocket</groupId>
        <artifactId>javax.websocket-api</artifactId>
        <version>1.0-rc3</version>
    </dependency>

    <dependency>
        <groupId>org.glassfish.tyrus</groupId>
        <artifactId>tyrus-client</artifactId>
        <version>1.0-rc3</version>
    </dependency>

    <dependency>
        <groupId>org.glassfish.tyrus</groupId>
        <artifactId>tyrus-server</artifactId>
        <version>1.0-rc3</version>
    </dependency>

    <dependency>
        <groupId>org.glassfish.tyrus</groupId>
        <artifactId>tyrus-container-grizzly</artifactId>
        <version>1.0-rc3</version>
    </dependency>
    

    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.2.0</version>
    </dependency>

    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-annotations</artifactId>
      <version>2.2.0</version>
    </dependency>

    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-core</artifactId>
      <version>2.2.0</version>
    </dependency>

  </dependencies>

So the first things we need to do is to define an implementations of the Encode/Decoder interfaces to do this work for us. This is going to do some simple reflection to workout what the bean class is. Like with JAX-WS it is easier to put them on the same class. Note that we use the streaming version of the interface and are only handling text content. (Ignoring the ability to send binary data for the moment)

package websocket;

import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.IOException;
import java.io.Reader;
import java.io.Writer;

import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;

import javax.websocket.DecodeException;
import javax.websocket.Decoder;
import javax.websocket.EncodeException;
import javax.websocket.Encoder;
import javax.websocket.EndpointConfig;

public abstract class JSONCoder<T>
  implements Encoder.TextStream<T>, Decoder.TextStream<T>{


    private Class<T> _type;
    
    // When configured my read in that ObjectMapper is not thread safe
    //
    private ThreadLocal<ObjectMapper> _mapper = new ThreadLocal<ObjectMapper>() {

        @Override
        protected ObjectMapper initialValue() {
            return new ObjectMapper();
        }
    };
    

    @Override
    public void init(EndpointConfig endpointConfig) {
    
        ParameterizedType $thisClass = (ParameterizedType) this.getClass().getGenericSuperclass();
        Type $T = $thisClass.getActualTypeArguments()[0];
        if ($T instanceof Class) {
            _type = (Class<T>)$T;
        }
        else if ($T instanceof ParameterizedType) {
            _type = (Class<T>)((ParameterizedType)$T).getRawType();
        }
    }

    @Override
    public void encode(T object, Writer writer) throws EncodeException, IOException {
        _mapper.get().writeValue(writer, object);
    }

    @Override
    public T decode(Reader reader) throws DecodeException, IOException {
        return _mapper.get().readValue(reader, _type);
    }

    @Override
    public void destroy() {
        
    }

}

The bean class is really quite simple with a static subclass of the Coder that we can use later.

package websocket;

public class EchoBean {
    
    
    public static class EchoBeanCode extends
       JSONCoder<EchoBean> {
        
    }
    
    
    private String _message;
    private String _reply;


    public EchoBean() {
        
    }

    public EchoBean(String _message) {
        super();
        this._message = _message;
    }


    public void setMessage(String _message) {
        this._message = _message;
    }

    public String getMessage() {
        return _message;
    }


    public void setReply(String _reply) {
        this._reply = _reply;
    }

    public String getReply() {
        return _reply;
    }

}

So new we need to implement our server endpoint, you can go one of two way either annotating a POJO or extending Endpoint. I am going with the first for the server and the second for the client. Really all this service does is to post the message back to the client. Note the registration of the encode and decoder. The same class in this case.

package websocket;

import java.io.IOException;

import javax.websocket.EncodeException;
import javax.websocket.EndpointConfig;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;
import static java.lang.System.out;

@ServerEndpoint(value="/echo",
                encoders = {EchoBean.EchoBeanCode.class},
                decoders = {EchoBean.EchoBeanCode.class})
public class EchoBeanService
{
    
    @OnMessage
    public void echo (EchoBean bean, Session peer) throws IOException, EncodeException {
        //
        bean.setReply("Server says " + bean.getMessage());
        out.println("Sending message to client");
        peer.getBasicRemote().sendObject(bean);
    }

    @OnOpen
    public void onOpen(final Session session, EndpointConfig endpointConfig) {
        out.println("Server connected "  + session + " " + endpointConfig);
    }
}

Lets look at a client bean, this time extending the standard Endpoint class and adding a specific listener for a message. In this case when the message is received the connection is simply closed to make our test case simple. In the real world managing this connection would obviously be more complicated.

package websocket;

import java.io.IOException;

import javax.websocket.ClientEndpoint;
import javax.websocket.CloseReason;
import javax.websocket.EncodeException;
import javax.websocket.Endpoint;
import javax.websocket.EndpointConfig;
import javax.websocket.MessageHandler;
import javax.websocket.Session;

import static java.lang.System.out;

@ClientEndpoint(encoders = {EchoBean.EchoBeanCode.class},
                decoders = {EchoBean.EchoBeanCode.class})
public class EchoBeanClient 
  extends Endpoint
{
    public void onOpen(final Session session, EndpointConfig endpointConfig) {

        out.println("Client Connection open "  + session + " " + endpointConfig);
        
        // Add a listener to capture the returning event
        //
        
        session.addMessageHandler(new MessageHandler.Whole() {

            @Override
            public void onMessage(EchoBean bean) {
                out.println("Message from server : " + bean.getReply());
                
                out.println("Closing connection");
                try {
                    session.close(new CloseReason(CloseReason.CloseCodes.NORMAL_CLOSURE, "All fine"));
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        });
        
        // Once we are connected we can now safely send out initial message to the server
        //
        
        out.println("Sending message to server");
        try {
            EchoBean bean = new EchoBean("Hello");
            session.getBasicRemote().sendObject(bean);
        } catch (IOException e) {
            e.printStackTrace();
        } catch (EncodeException e) {
            e.printStackTrace();
        }

    }
}

Now running the WebSocket standalone is really quite straightforward with Tyrus, you simple instantiate a Server and start it. Be aware this starts daemon threads so you need to make sure if this is in a main method that you do something to keep the JVM alive.

import org.glassfish.tyrus.server.Server;

Server server = new Server("localhost", 8025, "/", EchoBeanService.class);
server.start();


So the client is relatively simple; but as we are doing the declarative method we need to explicitly register the encoders and decoders when registering the client class.

import javax.websocket.ClientEndpointConfig;
import javax.websocket.Decoder;
import javax.websocket.Encoder;
import javax.websocket.Session;

import org.glassfish.tyrus.client.ClientManager;


// Right now we have to create a client, which will send a message then close
// when it has received a reply
//

ClientManager client = ClientManager.createClient();
EchoBeanClient beanClient = new EchoBeanClient();

Session session = client.connectToServer(
        beanClient, 
        ClientEndpointConfig.Builder.create()
         .encoders(Arrays.<Class<? extends Encoder>>asList(EchoBean.EchoBeanCode.class))
         .decoders(Arrays.<Class<? extends Decoder>>asList(EchoBean.EchoBeanCode.class))
         .build(),
        URI.create("ws://localhost:8025/echo"));
        
        
// Wait until things are closed down
        
while (session.isOpen()) {
    out.println("Waiting");
    TimeUnit.MILLISECONDS.sleep(10);
}

Now the output of this looks like the following:

Server connected SessionImpl{uri=/echo, id='e7739cc8-1ce5-4c26-ad5f-88a24c688799', endpoint=EndpointWrapper{endpointClass=null, endpoint=org.glassfish.tyrus.core.AnnotatedEndpoint@1ce5bc9, uri='/echo', contextPath='/'}} javax.websocket.server.DefaultServerEndpointConfig@ec120d
Waiting
Client Connection open SessionImpl{uri=ws://localhost:8025/echo, id='7428be2b-6f8a-4c40-a0c4-b1c8b22e1338', endpoint=EndpointWrapper{endpointClass=null, endpoint=websocket.EchoBeanClient@404c85, uri='ws://localhost:8025/echo', contextPath='ws://localhost:8025/echo'}} javax.websocket.DefaultClientEndpointConfig@15fdf14
Sending message to server
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Sending message to client
Message from server : Server says Hello
Closing connection
Waiting

Interestingly the first time this is run the there is a pause, I suspect this is due to Jackson setting itself up but I haven't had time to profile. I did find that this long delay on occurred on the first post - although obviously this is going to be slower than just passing plain text messages in general. Whether the different is significant to you depends on your application.

It would be interesting to compare the performance of the plain text with a JSON stream API such as that provided by the new JSR and of course the version that binds those values to a JSON POJO. Something for another day perhaps.

Thursday, April 25, 2013

Building a simple java client for forecast.io

I like to be outdoors when life allows so weather is always an interest for me. So i was very pleases to see forecast.io becoming available with global coverage after being jealous of the US only "DarkSkys" app. It was also nice to see that there is a nice REST api for the service which I could play with.

Unfortunately there was only a textual description of service, nothing I could consume with the Java client tools I have been working with. But I though it would be an interesting experiment to try to document an existing service from scratch.

The first thing I did was to get an example of the data that the service would respond with and run it through a JSON->JSON Schema tool. I used JSONSchema.net which gave me a rough v3 JSON Schema; but it wasn't able to take into account elements that are reused. A little bit of hand editing later I ended up with three JSON Schema files, one for the forecast itself and one for a list of weather details and and one for each particular data point.

From there is was pretty simple to read the developer documentation and write up a simple WADL to describe the two resources provided by the service. Luckily Jason LaPort from the DarkSkys was on hand to fix a few mistake and generally verify my description.

Once I had this all in place I of course found a couple of bugs in my own code, so it was time to update and fix wadl.java.net and release a 1.1.5 version with a revised depdency on the jsonschema2pojo project. This is why a little bit of adhoc expert testing pay dividends.

So taking the WADL from github it is possible to generate a nice clean Java API using the wadl.java.net generator. I won't reproduce the steps to generate the client here, but this is the end result and the kind of code you can get out of it.

// Create a Jersey client and setup for POJO JSON mapping

    ClientConfig cc = new DefaultClientConfig();
    cc.getFeatures().put(JSONConfiguration.FEATURE_POJO_MAPPING, true);
    Client c = Client.create(cc);


    // Get current weather for location and check for any alerts 
    
    Forecast f = ApiForecastIo_Forecast
        .apikey(c, "xxxxxxx")                  // Get own key from developer.forecast.io
        .latitudeLongitude(51.32d, -1.244d)    // Some location near where I live
           .getAsForecast(
              null,                            // Not JSONP
              Units.SI,                        // Return values in SI units
              "hourly,minutely,daily,flags");  // Just return current weather and any alerts
    
    // Output a summary

    System.out.println(
        f.getCurrently().getSummary() + " " +
        f.getCurrently().getTemperature() + " C");

    // Output any alerts
  
    List<Alert> alerts = f.getAlerts();
    for (Alert altert : alerts) {
        System.out.println(
            altert.getTitle() + " " + altert.getUri());
    }

The nice thing here was just how pleasantly easy describing a service that already exists was, yes I could have written it by hand in Java; but then when I wanted to move to Jersey 2.x I would have to re-write by hand rather that just regenerating from the metadata to a new technology target. (Also did the same for a BBC REST Schedule API, that is in the same git project)

Friday, March 22, 2013

wadl2java 1.1.4 released

We have released a new version of the wadl2java tool today with a bunch of new features and bug fixes. Again thanks for Pavel Bucek from the Jersey team for his continual help, and for the contribution from my JDeveloper team mate Michael Bachand.

New Features:

  • Added support for generating binding classes for WADL & JSON Schema decribed services.
  • Refactored upgrade transform to wadl-xslt module and added a WADL->HTML transform.
  • Added support for constructors that take a URI, this require use of an internal class UriTemplate in Jersey 2.x to have this work in the RS 2.0 version breaking general compatibility.
  • Fixes with regards to the requirement to use a later version of JAX-B than provided with JDK 6, some examples now required endorsed directory to run / build.
  • The default message for this exception with just a status was just an empty string, so we generate some boilerplate to fix this. (See JAX_RS_SPEC-312)

WADL Bugs:

  • WADL-65 Exception if non-required query parameters are passed in as null.
  • WADL-66 If the XML Schema didn't start with <?xml it would be treated as a JSON Schema.
  • WADL-52 UriBuilder impl dependency missing.

Oracle Bugs:

  • 16234340 matrix parameters were not showing up in the test URL because of inconsistent usage of the resourceTypes and the resourceObjects structures.
  • 16429792, where you would get duplicate methods if there were matrix parameters. These are dealt with like template parameters and as such are dealt with differently and needed to be filtered from the optional list.
  • 16069855 we are modifying the WADL stylesheet (wadl_2009-02.xsl) since it's resulting XHTML document is now properly depicting non ascii characters (such as a french accent) the fix involves adding a element specifying UTF-8 as the charset.
  • 16061180 Cannot process WADL with regex in template parameters, templates needed to be take out for code gen.
  • 15878417 Generate template method to make it easier for IDE tooling to customize client creation
  • 14831111 The client was no longer throwing an exception for a server side failure.
  • 14710107 where the generated javadoc would would not be rendered properly do a XML copying issue.

Future Work:

Going forward we are looking to generate "java.lang.reflect.Proxy" based clients based on the Jersey 2.x module or a future part of the JAX-RS 2.x specification. We are also going to look at transforming other REST service descriptions into XML, for example Swagger, so that they can be fed into the client generation tool.

Friday, February 22, 2013

Using UriTemplate to change path parameters in a URI

I have been recently working on a client generator in the wadl.java.net project and I was solving a bug where I wanted to change path parameter on a URI. This turned out to be quite easy with existing classes that are part of the Jersey project.

First of all you need a instance of UriTemplate, depending on the very of Jersey this is in a slightly different package - you IDE's automatic import will do the work for you.

  String template = "http://example.com/name/{name}/age/{age}";
  UriTemplate uriTemplate = new UriTemplate(template);


Then you use match(...) to extract them:

  String uri = "http://example.com/name/Bob/age/47";
  Map<String, String> parameters = new HashMap<>();

  // Not this method returns false if the URI doesn't match, ignored
  // for the purposes of the this blog.
  uriTemplate.match(uri, parameters);

Then you can replace the parameters at will and rebuild the URI:

  parameters.put("name","Arnold");

  UriBuilder builder = UriBuilder.fromPath(template);
  URI output = builder.build(parameters);

A little temporal feedback when running tests

I have been working a lot recently with a bunch of test jobs that are intermittently getting stuck. Now it is possible to work out the point where the problem is occurring by looking at the date labels on the LOG output; but it is hard visually to pick these discontinuities out.

In order improve the output in a way that my brain can really quickly parse and identify I thought I would try my hand a bit of simple ASCII animation. I wrote a simple fixture that writes out a line every 1 minute while the test is running, here is a very simple implementation - I am sure there are better ways to do the animation!

import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.TimeUnit;

public class TickTockFixture
{
   private Timer timer;
   

   @Before
   public void setUp()
   {
      // Clean out any old timers
      tearDown();
      
      // Start a new timer
      
      timer = new Timer();
      timer.scheduleAtFixedRate(new TimerTask()
      {
         int counter = 0;
         
         @Override
         public void run()
         {
            StringBuilder sb = new StringBuilder(". TickTock : ");
            int animation = counter++ % 10;
            boolean direction = animation < 5;
            int count = direction ?  animation : 9 - animation;
            sb.append("     ", 0, count);
            sb.append(direction ? "/" : "\\");
            sb.append("     ",0, 5-count);
            sb.append(counter);
            sb.append(" min(s)");
            System.err.println(sb);
         }
      }, 0, TimeUnit.MINUTES.toMillis(1));
      
   }

   @After
   public void tearDown()
   {
      if (timer!=null)
      {
         timer.cancel();
         timer = null;
      }
   }
}

So does this help, well take a look at the - made up example - output below and try to spot the section of the log output where everything slows down. It is a lot easier with the ticking output which I think sticks out better because of the animation.
22-Feb-2013 11:54:30 Items in drs, going to tidy up
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/weblogic.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/web.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/Hello.class
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/.module_marker
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/weblogic-application.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/application.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/.adrs-module.properties
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/
22-Feb-2013 11:54:53 oracle.jdevimpl.webservices.tcpmonitor.config.AnalyzerInstance stop
WARNING: AnalyzerInstance aborting a stop as already appears to be stopped
22-Feb-2013 11:54:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
22-Feb-2013 11:54:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
trigger seeding of SecureRandom
. TickTock :  \    9 min(s)
. TickTock : \     10 min(s)
. TickTock : /     11 min(s)
. TickTock :  /    12 min(s)
. TickTock :   /   13 min(s)
22-Feb-2013 11:59:50 done seeding SecureRandom
22-Feb-2013 11:59:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/Hello.class
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/

Monday, February 11, 2013

Proxying a request to localhost in Java

It can be very convenient when developing to server based application to run them using "localhost" in order to maintain consistency between developer machines. This is normally a good idea but there is a small case where this can cause problems.

Consider if you are running a local http proxy on your machine in order to capture your HTTP traffic. (Cough perhaps even the one in JDeveloper I work on). Then you might run into Java bug 6737819. Basically by default JDK 1.6 was hard coded not to send any request to localhost via a proxy which of course was a bit of a pain. Luckily a workaround was put in where you could put the string "~localhost" in your nonProxyHosts entry to turn of this feature:

java -client -classpath classes -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8099 -Dhttp.nonProxyHosts=~localhost -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8099 client.Example

Now moving forward to JDK 1.7 this workaround no longer works; but you need to take care to define nonProxyHosts as an empty string:

java -client -classpath classes -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8099 -Dhttp.nonProxyHosts= -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8099 client.Example

If you define this any anything other than an empty string the DefaultProxySelector though beware because internally it will append / or use the http.nonProxyHosts value from ../jre/lib/net.properties".

Just a minor complication that is not obvious from the published API.