Monday, November 17, 2014

But that's impossible, or finding out that the JIT has broken your code.

Every now and then you look at some code and think that it cannot be possibly be wrong. Once you have ruled out a simple programmer screw up / enemy action in code (Make sure you read Java Puzzlers or similar) or a concurrency issue (Read Java Concurrency or go on Dr Heniz excellent course) you should sit back and take a few days and then starting thinking about whether the JDK is indeed out to get you. I haven't seen one in the wild in my 18 odd years as a Java programmer so it kinda took me by surprise.

If you are running against JDK 8 in a large scale Swing application you might eventually see the following exception, lots of lots of times. (Unless you forgot the lesson learned in my previous blog in your logging code, in which case you might see lots of ArrayOfOutBoundsException)

Caused by: java.lang.NullPointerException 
    at javax.swing.text.GlyphView.getBreakSpot(GlyphView.java:799) 
    at javax.swing.text.GlyphView.getBreakWeight(GlyphView.java:724) 
    at javax.swing.text.FlowView$LogicalView.getPreferredSpan(FlowView.java:733) 
    at javax.swing.text.FlowView.calculateMinorAxisRequirements(FlowView.java:233) 
    at javax.swing.text.ParagraphView.calculateMinorAxisRequirements(ParagraphView.java:717) 
    at javax.swing.text.BoxView.checkRequests(BoxView.java:935) 
    at javax.swing.text.BoxView.getMinimumSpan(BoxView.java:568) 
    at javax.swing.text.BoxView.calculateMinorAxisRequirements(BoxView.java:903) 
    at javax.swing.text.BoxView.checkRequests(BoxView.java:935) 
    at javax.swing.text.BoxView.setSpanOnAxis(BoxView.java:343) 
    at javax.swing.text.BoxView.layout(BoxView.java:708) 
    at javax.swing.text.BoxView.setSize(BoxView.java:397) 
    ...

This error is particularly insidious because it takes around ten minutes to show itself and sometimes not at all. If you look at the code for this class the line in question, starts with "startsFrom = break", only accesses two local variables both of which have been previously referenced in the method.

            Segment s = getText(pstart, pend); 
            s.first(); 
            BreakIterator breaker = getBreaker(); 
            breaker.setText(s); 

            // Backward search should start from end+1 unless there's NO end+1 
            int startFrom = end + (pend > end ? 1 : 0); 
            for (;;) { 
                startFrom = breaker.preceding(s.offset + (startFrom - pstart))   
                          + (pstart - s.offset); 
                if (startFrom > start) { 
                    // The break spot is within the view 
                    bs[ix++] = startFrom; 
                } else { 
                    break; 
                } 
            } 

The most direct way to rule out a JIT error is to disable compilation for just this single method, here is an example; but you can fine more in the documentation for the command line java tool.

javaThing -XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot

When this parameter is added, the problem goes away. - since we have ruled out enemy action by code or a concurrency issue we can be more sure this is a JIT issue. Now as part of the bug logging for this I output diagnostics for this single method, and found out that the problem, didn't occur until the method was JITted for a fifth time.

javaThing -XX:CompileCommand=print,javax/swing/text/GlyphView,getBreakSpot

Here is some diagnostic output seen with the above command:

Compiled method (c2)  914078 33142       4       javax.swing.text.GlyphView::getBreakSpot (247 bytes)
 total in heap  [0x00002aaab0749e10,0x00002aaab0750fe0] = 29136
 relocation     [0x00002aaab0749f38,0x00002aaab074a1e8] = 688
 constants      [0x00002aaab074a200,0x00002aaab074a2a0] = 160
 main code      [0x00002aaab074a2a0,0x00002aaab074cde0] = 11072
 stub code      [0x00002aaab074cde0,0x00002aaab074ce40] = 96
 oops           [0x00002aaab074ce40,0x00002aaab074ce58] = 24
 metadata       [0x00002aaab074ce58,0x00002aaab074d058] = 512
 scopes data    [0x00002aaab074d058,0x00002aaab074ea20] = 6600
 scopes pcs     [0x00002aaab074ea20,0x00002aaab0750c50] = 8752
 dependencies   [0x00002aaab0750c50,0x00002aaab0750c80] = 48
 handler table  [0x00002aaab0750c80,0x00002aaab0750e90] = 528
 nul chk table  [0x00002aaab0750e90,0x00002aaab0750fe0] = 336
OopMapSet contains 113 OopMaps
#0 
OopMap{[8]=Oop [32]=Oop [40]=Oop off=892}
#1 
OopMap{[32]=Oop [40]=Oop off=960}
#2 
OopMap{[32]=Oop [40]=Oop off=980}
#3 
OopMap{[32]=Oop [40]=Oop [48]=Oop off=1048}
#4 
OopMap{[32]=Oop [40]=Oop [48]=Oop off=1084}
#5 
OopMap{[0]=Oop [24]=Oop [48]=Oop [56]=Oop [80]=Oop off=2500}
#6 
OopMap{rbx=Oop rdi=Oop [32]=Oop [40]=Oop [112]=Oop off=2533}
#7 
OopMap{rbx=Oop rdi=Oop r14=Oop [32]=Oop [112]=Oop off=3081}
#8 
OopMap{rbx=Oop rdi=Oop r14=Oop [32]=Oop [40]=Oop [112]=Oop off=3190}
#9 
OopMap{[8]=Oop [32]=Oop [40]=Oop off=4408}
#10 
OopMap{[32]=Oop [40]=Oop [48]=Oop off=4640}
#11 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=5232}
#12 
OopMap{rbp=Oop [0]=NarrowOop [32]=Oop off=5364}
#13 
OopMap{[32]=Oop [40]=Oop [48]=Oop off=5408}
#14 
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop off=5436}
#15 
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop off=5468}
#16 
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop off=5524}
#17 
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop [88]=Oop off=5552}
#18 
OopMap{[32]=Oop [40]=Oop [48]=Oop [64]=Oop [72]=Derived_oop_[64] [112]=Oop off=5608}
#19 
OopMap{[8]=Oop [32]=Oop off=5680}
#20 
OopMap{rbp=Oop off=5720}
#21 
OopMap{rbp=Oop off=5752}
#22 
OopMap{rbp=Oop [24]=NarrowOop [28]=NarrowOop [32]=Oop [40]=Oop [48]=Oop [56]=Oop [64]=Oop [88]=Oop off=5812}
#23 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop [88]=Oop off=5960}
#24 
OopMap{[0]=Oop [24]=Oop [48]=Oop [56]=Oop [72]=Oop [88]=NarrowOop off=6056}
#25 
OopMap{[40]=Oop off=6088}
#26 
OopMap{[0]=Oop off=6120}
#27 
OopMap{[8]=Oop [24]=Oop [56]=Oop [72]=Oop [112]=Oop off=6216}
#28 
OopMap{[0]=Oop [32]=NarrowOop [40]=Oop off=6284}
#29 
OopMap{rbp=Oop [16]=Oop [40]=Oop [64]=Oop [112]=Oop off=6384}
#30 
OopMap{[0]=Oop off=6412}
#31 
OopMap{[0]=Oop [16]=Oop [32]=NarrowOop [40]=Oop [48]=Oop off=6488}
#32 
OopMap{rbp=Oop [16]=Oop [40]=Oop [48]=Oop off=6560}
#33 
OopMap{[32]=Oop [40]=Oop [48]=Oop [64]=Oop [112]=Oop off=6608}
#34 
OopMap{[8]=Oop [28]=NarrowOop [32]=Oop [40]=Oop [48]=Oop off=6768}
#35 
OopMap{rbp=NarrowOop [0]=Oop [16]=Oop [32]=Oop [40]=NarrowOop off=6860}
#36 
OopMap{[0]=Oop [16]=Oop [32]=NarrowOop [40]=Oop [48]=Oop off=6988}
#37 
OopMap{rbp=Oop [32]=Oop off=7024}
#38 
OopMap{rbp=NarrowOop [0]=Oop [24]=Oop [32]=Oop off=7260}
#39 
OopMap{rbp=NarrowOop [0]=Oop [24]=Oop [32]=Oop off=7344}
#40 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [60]=NarrowOop [64]=Oop off=7452}
#41 
OopMap{rbp=NarrowOop [32]=Oop off=7476}
#42 
OopMap{rbp=NarrowOop [0]=Oop off=7524}
#43 
OopMap{[32]=Oop [40]=Oop [48]=Oop off=7588}
#44 
OopMap{[32]=Oop [40]=Oop [48]=Oop off=7616}
#45 
OopMap{[32]=Oop [40]=Oop [48]=Oop off=7632}
#46 
OopMap{rbp=NarrowOop [32]=Oop off=7676}
#47 
OopMap{rbp=NarrowOop [0]=Oop off=7724}
#48 
OopMap{[0]=Oop [16]=Oop [28]=NarrowOop [40]=Oop [48]=Oop [56]=NarrowOop [64]=Oop off=7868}
#49 
OopMap{[8]=Oop [28]=NarrowOop [32]=Oop [40]=Oop [48]=Oop [56]=Oop off=7916}
#50 
OopMap{rbp=Oop [16]=Oop [24]=NarrowOop off=8016}
#51 
OopMap{rbp=Oop [16]=Oop [28]=NarrowOop off=8080}
#52 
OopMap{rbp=NarrowOop [0]=Oop [24]=Oop [32]=Oop off=8152}
#53 
OopMap{rbp=Oop [8]=NarrowOop off=8212}
#54 
OopMap{rbp=NarrowOop [32]=Oop off=8236}
#55 
OopMap{rbp=Oop [16]=NarrowOop off=8272}
#56 
OopMap{rbp=NarrowOop [0]=Oop off=8320}
#57 
OopMap{rbp=Oop [12]=NarrowOop off=8360}
#58 
OopMap{rbp=NarrowOop [32]=Oop off=8400}
#59 
OopMap{rbp=Oop [12]=NarrowOop off=8460}
#60 
OopMap{rbp=NarrowOop [0]=Oop off=8508}
#61 
OopMap{rbp=Oop [24]=NarrowOop [40]=Oop off=8572}
#62 
OopMap{rbp=Oop off=8600}
#63 
OopMap{rbp=Oop [8]=Oop [28]=NarrowOop off=8640}
#64 
OopMap{rbp=Oop [8]=Oop [20]=NarrowOop [112]=Oop off=8704}
#65 
OopMap{rbp=Oop [16]=Oop [24]=Oop [48]=Oop off=8788}
#66 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=8912}
#67 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9036}
#68 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9160}
#69 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9284}
#70 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9408}
#71 
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9532}
#72 
OopMap{off=9556}
#73 
OopMap{off=9580}
#74 
OopMap{off=9604}
#75 
OopMap{[112]=Oop off=9628}
#76 
OopMap{rbp=Oop [8]=Oop [24]=Oop [32]=NarrowOop off=9696}
#77 
OopMap{rbp=Oop [8]=Oop [24]=NarrowOop off=9760}
#78 
OopMap{off=9784}
#79 
OopMap{off=9812}
#80 
OopMap{off=9836}
#81 
OopMap{off=9860}
#82 
OopMap{off=9884}
#83 
OopMap{off=9908}
#84 
OopMap{off=9932}
#85 
OopMap{off=9956}
#86 
OopMap{off=9980}
#87 
OopMap{off=10004}
#88 
OopMap{off=10028}
#89 
OopMap{rbp=Oop [16]=Oop [28]=NarrowOop off=10092}
#90 
OopMap{rbp=Oop [16]=Oop [24]=Oop [48]=Oop off=10176}
#91 
OopMap{off=10200}
#92 
OopMap{off=10224}
#93 
OopMap{off=10248}
#94 
OopMap{off=10272}
#95 
OopMap{off=10296}
#96 
OopMap{off=10320}
#97 
OopMap{off=10344}
#98 
OopMap{off=10368}
#99 
OopMap{off=10392}
#100 
OopMap{off=10416}
#101 
OopMap{off=10440}
#102 
OopMap{off=10464}
#103 
OopMap{off=10488}
#104 
OopMap{off=10512}
#105 
OopMap{off=10536}
#106 
OopMap{off=10560}
#107 
OopMap{off=10584}
#108 
OopMap{off=10608}
#109 
OopMap{off=10632}
#110 
OopMap{off=10656}
#111 
OopMap{off=10680}
#112 
OopMap{off=11028}
java.lang.NullPointerException
 at javax.swing.text.GlyphView.getBreakSpot(GlyphView.java:799)
 at javax.swing.text.GlyphView.getBreakWeight(GlyphView.java:724)
 at javax.swing.text.html.InlineView.getBreakWeight(InlineView.java:150)
 at javax.swing.text.FlowView$LogicalView.getPreferredSpan(FlowView.java:733)
 at javax.swing.text.FlowView.calculateMinorAxisRequirements(FlowView.java:233)
 at javax.swing.text.ParagraphView.calculateMinorAxisRequirements(ParagraphView.java:717)
 at javax.swing.text.html.ParagraphView.calculateMinorAxisRequirements(ParagraphView.java:157)
 at javax.swing.text.BoxView.checkRequests(BoxView.java:935)
 at javax.swing.text.BoxView.getMinimumSpan(BoxView.java:568)
 at javax.swing.text.html.ParagraphView.getMinimumSpan(ParagraphView.java:270)
 at javax.swing.text.BoxView.calculateMinorAxisRequirements(BoxView.java:903)

Now I am still working with the JDK team for a fix for this one; but I do feel I have discovered a useful set of tools for providing some evidence that the JIT compiler is causing my bad day. And more importantly I have a workaround so I can run my tests until this is resolved.

Thursday, November 6, 2014

Missing stack traces for repeated exceptions (reminder)

A long while ago a optimisation was added to the JVM so that if the same exception is thrown again and again and again a single instance of the Exception is created without the stack trace filled in in order to increase performance. This is an excellent idea unless you are trying to diagnose a problem and you have missed the original error.

If you forgot about this optimisation you send the afternoon looking at the following log output and weeping slightly. (In my defence I have a little one in the house hence the fuzzy brain and lack of blogging action this year)

java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
...
java.lang.Exception: Uncaught exceptions during test
 at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/common-adf/CreateNewCustomAppAndProjectWithName.xml:25)
 at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/level0/testADFcMATS.xml:94)
 at oracle.abbot.JDevScriptFixture.runTest(JDevScriptFixture.java:555)
 at junit.framework.TestCase.runBare(TestCase.java:134)
 at junit.framework.TestResult$1.protect(TestResult.java:110)
 at junit.framework.TestResult.runProtected(TestResult.java:128)
 at junit.framework.TestResult.run(TestResult.java:113)
 at junit.framework.TestCase.run(TestCase.java:124)
 at junit.framework.TestSuite.runTest(TestSuite.java:243)
 at junit.framework.TestSuite.run(TestSuite.java:238)
 at junit.textui.TestRunner.doRun(TestRunner.java:116)
 at junit.textui.TestRunner.doRun(TestRunner.java:109)
 at oracle.abbot.AbbotRunner.run(AbbotRunner.java:614)
 at oracle.abbot.AbbotAddin$IdeAbbotRunner.run(AbbotAddin.java:634)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException

[Crickets]

It turns out that you can turn off this optimisation with a simple flag:

java -XX:-OmitStackTraceInFastThrow ....

In my particular case the actual exception causing this trouble was a NPE from the GlyphView code in JDK8. (One that is being caused by a glitch in hotspot it seems) But that in turn was causing the AIOOBE in some logging code clouding the issue even more.

This in particular is a good flag to add by default when running your automated tests, particularly in combination with the stack trace length override I have talked about before.

Wednesday, May 21, 2014

Reading and writing JAX-RS Link objects

JAX-RS contains a rather nice handy representation of the a Link that can be serialised with and adapter into XML and JSON, unfortunately there is a bug in the JAX-RS spec that means that the standard adapter provided is missing the setters required to deserialise the Link later on.

This isn't a problem if you are using the JAX-B RI as it appears to be more relaxed than the standard; but it will be a problem for other implementations. There is a further bug if you are using MOXy, aka EclipseLink, to produce either JSON or XML that it will fail and just call toString() because it doesn't like the type adapter being an inner class of the class that is being adapted. (Bug TBC)


Direction JAXB RI + JAX-RS Adapter MOXy + JAX-RS Adapter MOXy + Outer Adapter MOXy + Outer Adapter + Setters MOXy + Inner Adapter (Different class)
Marshalling Yes No Yes Yes Yes
Unmarshalling Yes No No Yes No


The workaround is to create a new adapter as a top level class in order to replace the one provided by the standard. Just quickly it would look something like this:

import java.util.HashMap;
import java.util.Map;

import javax.ws.rs.core.Link;

import javax.xml.bind.annotation.adapters.XmlAdapter;
import javax.xml.namespace.QName;

public class LinkAdapter
    extends XmlAdapter {

    public LinkAdapter() {
    }

    public Link unmarshal(LinkJaxb p1) {
        
        Link.Builder builder = Link.fromUri(p1.getUri());
        for (Map.Entry<QName, Object> entry : p1.getParams().entrySet()) {
            builder.param(entry.getKey().getLocalPart(), entry.getValue().toString());
        }
        return builder.build();
    }

    public LinkJaxb marshal(Link p1) {
        
        Map<QName, Object> params = new HashMap<>();
        for (Map.Entry<String,String> entry : p1.getParams().entrySet()) {
            params.put(new QName("", entry.getKey()), entry.getValue());
        }
        return new LinkJaxb(p1.getUri(), params);
    }
}

With a simple POJO to go with it.

import java.net.URI;

import java.util.HashMap;
import java.util.Map;

import javax.xml.bind.annotation.XmlAnyAttribute;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.namespace.QName;

public class LinkJaxb  {

    private URI uri;
    private Map<QName, Object> params;


    public LinkJaxb() {
        this (null, null);
    }

    public LinkJaxb(URI uri) {
        this(uri, null);
    }

    public LinkJaxb(URI uri, Map<QName, Object> map) {
        
        this.uri = uri;
        this.params = map!=null ? map : new HashMap<QName, Object>();
        
    }



    @XmlAttribute(name = "href")
    public URI getUri() { 
        return uri;
    }

    @XmlAnyAttribute
    public Map<QName, Object> getParams() { 
        return params;
    }

    public void setUri(URI uri) {
        this.uri = uri;
    }

    public void setParams(Map<QName, Object> params) {
        this.params = params;
    }

}


With this you can read and write the Link objects to and from XML, and JSON with MOXy, to your hearts content.

Tuesday, May 20, 2014

Declarative Linking in Jersey 2.9 and up

A couple of weeks ago A couple of months ago I was looking how I was going to engineers new REST API for an Oracle Cloud project. Once of the things I had planned to do was to use the declarative link injection created in Jersey 1.x by Marc Hadley. Sadly this hadn't been forwarded ported yet, so a quick chat to the project lead and I took on the small medium sized job of bringing the code up to date.

One of the things that has changed in the new version is that in JAX-RS 2.0 there is a Link object so rather than being able to only inject String and URI you can also inject the correct rel attributes. This has means that the existing annotations coded by Marc have been merged into once simple set of annotations for both Link headers and for injected properties.

This functionality is available now, along with a simple example. The original version of the feature that I committed has some serious limitations that are described later, you will need a version of Jersey post 2.8 or you can build a 2.9-SNAPSHOT image that contains my changes currently to implement the example in this blog.

This blog looks at using this new API to provide simple injection for a collections API. One of the common patterns in RESTful services, in particular those based on JSON, is to have an array of structural links at the top level of the structure. For the purposes of this blog I am going to follow the form of the Collection+JSON hypermedia type.

{ "collection" :
  {
    "version" : "1.0",
    "href" : "http://example.org/friends/?offset=10&limit=10",
    
    "links" : [
      {"rel" : "create", "href" : "http://example.org/friends/"}
      {"rel" : "next", "href" : "http://example.org/friends/?offset=20&limit=10"}
      {"rel" : "previous", "href" : "http://example.org/friends/?offset=0&limit=10"}
    ],
   
    "items" : [
       ...
    ]
  }
}

So I can inject the links in the following form, not there is a bunch of boiler plate missing here for clarity. This is not the tidiest code; but in a later cycle it should be possible to simply them somewhat. The design currently uses EL to access properties - this has the advantage of making it possible to write back values as you can represent properties. I can understand it is disliked by some; but I am not sure if I see any value in moving to JavaScript at the moment. Also don't be put of by the @Xml annotations, I am using MOXy for JSON generation - this isn't an XML only thing.

{


  @XmlTransient
  private int limit, offset; // Getters for these

  @XmlTransient
  private int modelLimit; // Getters for these


  @InjectLink(
            resource = ItemsResource.class,
            method = "query",
            style = Style.ABSOLUTE,
            bindings = {@Binding(name = "offset", value="${instance.offset}"),
                @Binding(name = "limit", value="${instance.limit}")
            },
            rel = "self"
  )
  @XmlElement(name="link")
  private String href;

  @InjectLinks({
    @InjectLink(
          resource = ItemsResource.class,
          style = Style.ABSOLUTE,
          method = "query",
          condition = "${instance.offset + instance.limit < instance.modelLimit}",
          bindings = {
            @Binding(name = "offset", value = "${instance.offset + instance.limit}"),
            @Binding(name = "limit", value = "${instance.limit}")
          },
          rel = "next"
    ),
    @InjectLink(
          resource = ItemsResource.class,
          style = Style.ABSOLUTE,
          method = "query",
          condition = "${instance.offset - instance.limit >= 0}",
          bindings = {
            @Binding(name = "offset", value = "${instance.offset - instance.limit}"),
            @Binding(name = "limit", value = "${instance.limit}")
          },
          rel = "prev"
  )})
  @XmlElement(name="link")
  @XmlElementWrapper(name = "links")
  @XmlJavaTypeAdapter(Link.JaxbAdapter.class)
  List<Link> links;

  ....
}


The original porting of the declarative linking code that exists in version Jersey before 2.8 had very naive code with regards working out what the URI should be for a particular resource, it couldn't deal with any resources that were not at the root of the application, nor would it cope with query parameters which are so important when dealing with collections.

In theory there could be more than one URI for a particular resource class; but this code does need to assume a 1:1 mapping, the current implementation contains a simple algorithm that walks the Jersey meta-model to try to work out the structure, is this doesn't work in your can you can simple provide another implementation of ResourceMappingContext.

Some may question why should I use these ugly annotations when it might be easier just to inject the URI myself? Well the reason is to provide metadata that other tools can use. One of my next jobs is to extend this work to generate the hypermedia extensions and for this I need the above metadata. (Waiting on a pull request being approved before I can really get into it).

Finally it is worth noting that the paging model has its own problems which become apparent if you think of a REST collection as some kind of array that you can safely page over. Concurrent updates along with the lack of state mean that the client can never be sure they have the complete model and should expect to see some items more than once as the model is updated. Cursor or linking based schemes should be considered instead which is yet another good reminder as to why you should always treat the URI as opaque - the sever might need to changes its structure in the future. But that is a entirely different blog for another day.....

Update: See the following blog for some workarounds when reading and writing Link objects.

Tuesday, April 15, 2014

Quick, and a bit dirty, JSON Schema generation with MOXy 2.5.1

So I am working on a new REST API for an upcoming Oracle cloud service these days so one of the things I needed was the ability to automatically generate a JSON Schema for the bean in my model. I am using MOXy to generate the JSON from POJO and as of version 2.5.1 of EclipseLink it now has the ability to generate a JSON Schema from the bean model.

There will be a more formal solution integrated into Jersey 2.x at a future date; but this solution will do at the moment if you want to play around with this.

So the first class we need to put in place is a model processor, very much and internal Jersey class, that allows us to amend the resource model with extra methods and resources. To each resource in the model we can add the JsonSchemaHandler which does the hard work of generating a new schema. Since this is a simple POC there is no caching going on here, please be aware of this if you are going to use this in production code.

import com.google.common.collect.Lists;

import example.Bean;

import java.io.IOException;
import java.io.StringWriter;

import java.text.SimpleDateFormat;

import java.util.Date;
import java.util.List;

import javax.inject.Inject;

import javax.ws.rs.HttpMethod;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.core.Configuration;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import javax.xml.bind.JAXBException;
import javax.xml.bind.SchemaOutputResolver;
import javax.xml.transform.Result;
import javax.xml.transform.stream.StreamResult;

import org.eclipse.persistence.jaxb.JAXBContext;

import org.glassfish.jersey.process.Inflector;
import org.glassfish.jersey.server.ExtendedUriInfo;
import org.glassfish.jersey.server.model.ModelProcessor;
import org.glassfish.jersey.server.model.ResourceMethod;
import org.glassfish.jersey.server.model.ResourceModel;
import org.glassfish.jersey.server.model.RuntimeResource;
import org.glassfish.jersey.server.model.internal.ModelProcessorUtil;
import org.glassfish.jersey.server.wadl.internal.WadlResource;

public class JsonSchemaModelProcessor implements ModelProcessor {

  private static final MediaType JSON_SCHEMA_TYPE = 
    MediaType.valueOf("application/schema+json");
  private final List<ModelProcessorUtil.Method> methodList;


  public JsonSchemaModelProcessor() {
    methodList = Lists.newArrayList();
    methodList.add(new ModelProcessorUtil.Method("$schema", HttpMethod.GET, 
      MediaType.WILDCARD_TYPE, JSON_SCHEMA_TYPE,
      JsonSchemaHandler.class));
  }

  @Override
  public ResourceModel processResourceModel(ResourceModel resourceModel, 
      Configuration configuration) {
    return ModelProcessorUtil.enhanceResourceModel(resourceModel, true, methodList, 
      true).build();
  }

  @Override
  public ResourceModel processSubResource(ResourceModel resourceModel, 
      Configuration configuration) {
    return ModelProcessorUtil.enhanceResourceModel(resourceModel, true, methodList, 
      true).build();
  }


  public static class JsonSchemaHandler 
    implements Inflector<ContainerRequestContext, Response> {

    private final String lastModified = new SimpleDateFormat(WadlResource.HTTPDATEFORMAT).format(new Date());

    @Inject
    private ExtendedUriInfo extendedUriInfo;

    @Override
    public Response apply(ContainerRequestContext containerRequestContext) {

      // Find the resource that we are decorating, then work out the
      // return type on the first GET

      List<RuntimeResource> ms = extendedUriInfo.getMatchedRuntimeResources();
      List<ResourceMethod> rms = ms.get(1).getResourceMethods();
      Class responseType = null;
      found:
      for (ResourceMethod rm : rms) {
        if ("GET".equals(rm.getHttpMethod())) {
          responseType = (Class) rm.getInvocable().getResponseType();
          break found;
        }
      }

      if (responseType == null) {
        throw new WebApplicationException("Cannot resolve type for schema generation");
      }

      //
      try {
        JAXBContext context = (JAXBContext) JAXBContext.newInstance(responseType);

        StringWriter sw = new StringWriter();
        final StreamResult sr = new StreamResult(sw);

        context.generateJsonSchema(new SchemaOutputResolver() {
          @Override
          public Result createOutput(String namespaceUri, String suggestedFileName) 
              throws IOException {
            return sr;
          }
        }, responseType);


        return Response.ok().type(JSON_SCHEMA_TYPE)
          .header("Last-modified", lastModified)
          .entity(sw.toString()).build();
      } catch (JAXBException jaxb) {
        throw new WebApplicationException(jaxb);
      }
    }
  }


}

Note the very simple heuristic in the JsonSchemaHandler code it assumes that for each resource there is a 1:1 mapping to a single JSON Schema element. This of course might not be true for your particular application.

Now that we have the schema generated in a know location we need to tell the client about it, the first thing we will do is to make sure that there is a suitable link header when the user invokes OPTIONS on a particular resource:

import java.io.IOException;

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerResponseContext;
import javax.ws.rs.container.ContainerResponseFilter;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.Link;
import javax.ws.rs.core.UriInfo;

public class JsonSchemaResponseFilter implements ContainerResponseFilter {

  @Context
  private UriInfo uriInfo;

  @Override
  public void filter(ContainerRequestContext containerRequestContext,
                     ContainerResponseContext containerResponseContext) throws IOException {


    String method = containerRequestContext.getMethod();
    if ("OPTIONS".equals(method)) {

      Link schemaUriLink =
        Link.fromUriBuilder(uriInfo.getRequestUriBuilder()
          .path("$schema")).rel("describedBy").build();

      containerResponseContext.getHeaders().add("Link", schemaUriLink);
    }
  }
}

Since this is JAX-RS 2.x we are working with we of course are going bundle all the bit together into a feature:

import javax.ws.rs.core.Feature;
import javax.ws.rs.core.FeatureContext;

public class JsonSchemaFeature implements Feature {

  @Override
  public boolean configure(FeatureContext featureContext) {

    if (!featureContext.getConfiguration().isRegistered(JsonSchemaModelProcessor.class)) {
      featureContext.register(JsonSchemaModelProcessor.class);
      featureContext.register(JsonSchemaResponseFilter.class);
      return true;
    }
    return false;
  }
}

I am not going to show my entire set of POJO classes; but just quickly this is the Resource class with the @GET method required by the schema generation code:

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Path("/bean")
public class BeanResource {
    
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Bean getBean() {
        return new Bean();
    }
}

And finally here is what you see if you perform a GET on a resource:

GET .../resources/bean
Content-Type: application/json

{
  "message" : "hello",
  "other" : {
    "message" : "OtherBean"
  },
  "strings" : [
    "one",
    "two",
    "three",
    "four"
  ]
}

And OPTIONS:

OPTIONS .../resources/bean
Content-Type: text/plain
Link: <http://.../resources/bean/$schema>; rel="describedBy"

GET, OPTIONS, HEAD

And finally if you resolve the schema resource:

GET .../resources/bean/$schema
Content-Type: application/schema+json

{
  "$schema" : "http://json-schema.org/draft-04/schema#",
  "title" : "example.Bean",
  "type" : "object",
  "properties" : {
    "message" : {
      "type" : "string"
    },
    "other" : {
      "$ref" : "#/definitions/OtherBean"
    },
    "strings" : {
      "type" : "array",
      "items" : {
        "type" : "string"
      }
    }
  },
  "additionalProperties" : false,
  "definitions" : {
    "OtherBean" : {
      "type" : "object",
      "properties" : {
        "message" : {
          "type" : "string"
        }
      },
      "additionalProperties" : false
    }
  }
}

There is a quite a bit of work to do here, in particular generating the hypermedia extensions based on the declarative linking annotations that I forward ported into Jersey 2.x a little while back. But it does point towards a solution and we get to exercise a variety of solutions to get something working now.

Friday, February 7, 2014

Transparent PATCH support in JAX-RS 2.0

The PATCH method is one the the less well loved HTTP methods simple because until recently there really wasn't a standard PATCH format. This has been standardized for JSON for a while now so there are quite a few libraries that will do the heavy lifting for you. For the purposes of this blog I am going to use json-patch although it would be easy to adapt this particular implementation to the patch library of your choice.

A per normal lets get the resource and bean classes out of the way. In this example code we have a simple resource that knows how to return the original object and one that allows you to perform the PATCH method. Note that the patch method just accepts the bean object, this is because of some magic we are going to do in a little bit to pre-process the patch.

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Path("service")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class Service {

  @GET
  public Bean get() {
    return new Bean(true);
  }

  @PATCH
  @Consumes("application/json-patch+json")
  public Bean patch(Bean input) {
    System.out.println(input.getMessage() + "  " + input.getTitle());
    return input;
  }

}


import java.util.ArrayList;
import java.util.List;

public class Bean {

  private String title = "title";
  private String message = "message";
  private List<String> list = new ArrayList<String>();

  public Bean() {
    this(false);
  }

  public Bean(boolean init) {
    if (init) {
      title = "title";
      message = "message";
      list.add("one");
      list.add("two");
    }
  }


  public void setList(List list) {
    this.list = list;
  }

  public List getList() {
    return list;
  }

  public void setTitle(String title) {
    this.title = title;
  }

  public String getTitle() {
    return title;
  }

  public void setMessage(String message) {
    this.message = message;
  }

  public String getMessage() {
    return message;
  }

}

So the @PATCH annotation is something we have to create for this example, luckily JAX-RS contains a extension meta-annotation for this purpose. We are also going to use @NameBinding as this example is using JAX-RS 2.0 so we can connect up our filter in a moment.

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import javax.ws.rs.HttpMethod;
import javax.ws.rs.NameBinding;


@Target({ ElementType.METHOD, ElementType.TYPE })
@Retention(RetentionPolicy.RUNTIME)
@HttpMethod("PATCH")
@Documented
@NameBinding
public @interface PATCH {
}

So here is the implementation of the ReaderInterceptor that will process the incoming stream and replace it with the patched version. Note that the class is annotated with @PATCH also in order to make the @NamedBinding magic work and also that there is a lot of error handling that is missing as this is a simple POC.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import com.github.fge.jsonpatch.JsonPatch;
import com.github.fge.jsonpatch.JsonPatchException;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

import java.lang.annotation.Annotation;
import java.lang.reflect.Method;

import javax.ws.rs.GET;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedHashMap;
import javax.ws.rs.core.UriInfo;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
import javax.ws.rs.ext.ReaderInterceptor;
import javax.ws.rs.ext.ReaderInterceptorContext;

import org.glassfish.jersey.message.MessageBodyWorkers;

@Provider
@PATCH
public class PatchReader implements ReaderInterceptor {
  private UriInfo info;
  private MessageBodyWorkers workers;

  @Context
  public void setInfo(UriInfo info) {
    this.info = info;
  }

  @Context
  public void setWorkers(MessageBodyWorkers workers) {
    this.workers = workers;
  }

  @Override
  public Object aroundReadFrom(
    ReaderInterceptorContext readerInterceptorContext) 
    throws IOException,
           WebApplicationException {

    // Get the resource we are being called on, 
    // and find the GET method
    Object resource = info.getMatchedResources().get(0);

    Method found = null;
    for (Method next : resource.getClass().getMethods()) {
      if (next.getAnnotation(GET.class) != null) {
        found = next;
        break;
      }
    }


    if (found != null) {

      // Invoke the get method to get the state we are trying to patch
      //
      Object bean;
      try {
        bean = found.invoke(resource);
      } catch (Exception e) {
        throw new WebApplicationException(e);
      }
      
      // Convert this object to a an aray of bytes 
      ByteArrayOutputStream baos = new ByteArrayOutputStream();
      MessageBodyWriter<? super Object> bodyWriter =
        workers.getMessageBodyWriter(Object.class, bean.getClass(), 
          new Annotation[0], MediaType.APPLICATION_JSON_TYPE);

      bodyWriter.writeTo(bean, bean.getClass(), bean.getClass(), 
          new Annotation[0], MediaType.APPLICATION_JSON_TYPE,
          new MultivaluedHashMap<String, Object>(), baos);


      // Use the Jackson 2.x classes to convert both the incoming patch  
      // and the current state of the object into a JsonNode / JsonPatch
      ObjectMapper mapper = new ObjectMapper();
      JsonNode serverState = mapper.readValue(baos.toByteArray(), 
        JsonNode.class);
      JsonNode patchAsNode = mapper.readValue(
         readerInterceptorContext.getInputStream(), 
        JsonNode.class);
      JsonPatch patch = JsonPatch.fromJson(patchAsNode);

      try {
        // Apply the patch
        JsonNode result = patch.apply(serverState);

        // Stream the result & modify the stream on the readerInterceptor
        ByteArrayOutputStream resultAsByteArray = 
          new ByteArrayOutputStream();
        mapper.writeValue(resultAsByteArray, result);
        readerInterceptorContext.setInputStream(
          new ByteArrayInputStream(
            resultAsByteArray.toByteArray()));

        // Pass control back to the Jersey code
        return readerInterceptorContext.proceed();


      } catch (JsonPatchException e) {
        throw new WebApplicationException(
          Response.status(500).type("text/plain").entity(e.getMessage()).build());
      }

    } else {
      throw new IllegalArgumentException("No matching GET method on resource");
    }


  }
}

So once you have this deployed you can start playing with the data, so the original message is:

{
  "list" : [
    "one",
    "two"
  ],
  "message" : "message",
  "title" : "title"
}

So if you apply the following patch, the result returned is:

[
  {
    "op" : "replace",
    "path" : "/message",
    "value" : "otherMessage"
  },
  {
    "op" : "add",
    "path" : "/list/-",
    "value" : "three"
  }
]


{
  "list" : [
    "one",
    "two",
    "three"
  ],
  "message" : "otherMessage",
  "title" : "title"
}

This example shows it is relatively trivial to add PATCH support to your classes by following a simple coding pattern and using a simple Annotation. In this way PATCH support becomes trivial as the implementation can just delegate to your existing PUT method.

Update: Mirsolav Fuksa from the Jersey team reminded me that in order for this implementation to comply with the PATCH RFC it should provide the Accept-Patch header when the client performs an OPTIONS request. You can do this with a simple CotnainerResponseFilter:

import java.io.IOException;

import java.util.Collections;

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerResponseContext;
import javax.ws.rs.container.ContainerResponseFilter;
import javax.ws.rs.ext.Provider;

@Provider
public class OptionsAcceptHeader implements ContainerResponseFilter {

  @Override
  public void filter(ContainerRequestContext requestContext,
                     ContainerResponseContext responseContext) throws IOException {

    if ("OPTIONS".equals(requestContext.getMethod())) {
      if (responseContext.getHeaderString("Accept-Patch")==null) {
        responseContext.getHeaders().put(
          "Accept-Patch", Collections.<Object>singletonList("application/json-patch+json"));  
      }
    }
  }
}

Thursday, February 6, 2014

wadl2java 1.6 released

Just an incremental release with a cluster of bug fixes:

  • Upgrade to jsonschema2pojo 0.4.0
  • Fixed bugs where generation would fail due to duplicated class names caused by similar resource names
  • Upgrade Jersey 2-ALPHA deps to 2.5.1 some small changes in the APIs
  • Fix for XSL for correct return on onClick hander, allows better integration with IDE's
  • Interface of MethodNode.getSupportedOutput to support latest WADL model
  • Method parameter were not being shown consistently up the tree when using the XSLT
  • Reserved words were not being escaped when generating the client
  • Non-standard lines in the XSL causing failure on other non oracle transformers
  • WADL XSL now generates expandable / collapsable sections
Most of the work in this release was done by Michael Bachand who recently left Oracle, thanks very much for all of this efforts in this. (And of course Pavel for doing the release for me)