Working with XML and JSON

XML and JSON are the dominant formats to structure data that can be exchanged between parts of a system such as backend-frontend or between external systems. In Scala, there is some out-of-the-box support in the Scala library to manipulate both.

Manipulating XML

We have briefly seen earlier in this chapter as well as in Chapter 3, Understanding the Scala Ecosystem, when working with HTTP that XML documents can be created as literals and transformed in many ways. For instance, if we launch an REPL by typing > play console from a Play project root directory, we can start experimenting with XML:

scala> val books =
       <Library>
         <book title="Programming in Scala" quantity="15" price="30.00" />
         <book title="Scala for Java Developers" quantity="10" price="25.50" />
       </Library>
books: scala.xml.Elem = 
<Library>
  <book title="Programming in Scala" quantity="15" price="30.00"/>
  <book title="Scala for Java Developers" quantity="10" price="25.50"/>
</Library>

The books variable is of type Elem, which represents an XML structure. Rather than directly writing an XML literal, we could also construct the XML Elem using utility methods to parse a file or just parse a string, as follows:

scala> import scala.xml._
scala> val sameBooks = XML.loadString("""
         <Library>
           <book title="Programming in Scala" quantity="15" price="30.00"/>
           <book title="Scala for Java Developers" quantity="10" price="25.50"/>
         </Library>
       """)
sameBooks: scala.xml.Elem = 
<Library>
<book price="30.00" quantity="15" title="Programming in Scala"/>
<book price="25.50" quantity="10" title="Scala for Java Developers"/>
</Library>

The triple quote used in the preceding command lets us express a preformatted string where the characters are escaped (for example, the " within a regular string would have been noted \").

Processing such an XML structure can, for example, consist of computing the total price for the books. This operation can be achieved with a Scala for comprehension leading to the following code:

scala> val total = (for {
         book <- books \ "book"
         price = ( book \ "@price").text.toDouble
         quantity = ( book \ "@quantity").text.toInt
       } yield price * quantity).sum
total: Double = 705.0

Retrieving and transforming XML structures happens all the time when dealing with the integration of diverse external systems. Accessing the various XML tags through XPath expressions as we have done earlier is very handy and produces concise and readable code. Programmatically, creating XML out of information exported from Excel in the form of CSV data is also a common operation and can be achieved as follows:

scala> val books = 
       <Library>
         { List("Programming in Scala,15,30.00","Scala for Java Developers,10,25.50") map { row => row split "," } map { b => <book title={b(0)} quantity={b(1)} price={b(2)} /> }}
       </Library>
books: scala.xml.Elem = 
<Library>
  <book title="Programming in Scala" quantity="15" price="30.00"/><book title="Scala for Java Developers" quantity="10" price="25.50"/>
</Library>

Manipulating JSON

JSON is supported in the Scala library and you just need to import the appropriate library. An example of some REPL usage is illustrated as follows:

scala> import scala.util.parsing.json._
import scala.util.parsing.json._
scala> val result = JSON.parseFull("""
       {
         "Library": {
           "book": [
             {
               "title": "Scala for Java Developers",
               "quantity": 10
             },
             {
               "title": "Programming Scala",
               "quantity": 20
             }
           ]
         }
       }
       """)
result: Option[Any] = Some(Map(Library -> Map(book -> List(Map(title -> Scala for Java Developers, quantity -> 10.0), Map(title -> Programming Scala, quantity -> 20.0)))))

Any valid JSON message can be transformed into a structure made of Maps and Lists. However, it is often desirable to create meaningful classes, that is, expressing the business domain out of the JSON messages. The online service available at http://json2caseclass.cleverapps.io does exactly that; it is a convenient JSON to Scala case class converter. We can, for example, copy our preceding JSON message into the Json paste text area and click on the Let's go! button to try it out as shown in the following screenshot:

Manipulating JSON

The converter produces the following output:

case class Book(title:String, quantity:Double)
case class Library(book:List[Book])
case class R00tJsonObject(Library:Library)

Among the very interesting features of case classes that we have already introduced in Chapter 1, Programming Interactively within Your Project, is a decomposition mechanism for pattern matching. Once JSON messages have been deserialized into case classes, we can, for instance, manipulate them using this mechanism, as the sequence of the following command illustrates:

scala> case class Book(title:String, quantity:Double)
defined class Book
scala> val book1 = Book("Scala for Java Developers",10)
book1: Book = Book(Scala for Java Developers,10.0)
scala> val book2 = Book("Effective Java",12)
book2: Book = Book(Effective Java,12.0)
scala> val books = List(book1,book2)
books: List[Book] = List(Book(Scala for Java Developers,10.0), Book(Effective Java,12.0))

First, we defined two instances of books and put them into a list.

scala> def bookAboutScala(book:Book) = book match {
         case Book(a,_) if a contains "Scala" => Some(book)
         case _ => None
       }
bookAboutScala: (book: Book)Option[Book]

The method defined previously does pattern matching on the Book constructor, which also contains a guard (that is, the if condition). Since we do not use the second constructor parameter, we have put an underscore instead of creating an anonymous variable. Calling this method on both the book instances that we defined earlier will show the following result:

scala> bookAboutScala(book1)
res0: Option[Book] = Some(Book(Scala for Java Developers,10.0))
scala> bookAboutScala(book2)
res1: Option[Book] = None

We can mix case class pattern matching together with other patterns. Let's, for instance, define the following regular expression (note the usage of the triple quotes as well as the use of .r to specify that it is a regular expression):

scala> val regex = """(.*)(Scala|Java)(.*)""".r
regex: scala.util.matching.Regex = (.*)(Scala|Java)(.*)

This regular expression will match any string that contains either Scala or Java.

scala> def whatIs(that:Any) = that match {
         case Book(t,_) if (t contains "Scala") =>
           s"${t} is a book about Scala"
         case Book(_,_) => s"$that is a book "
         case regex(_,word,_) => s"$that is something about ${word}"
         case head::tail => s"$that is a list of books"
         case _ => "You tell me !"
       }
whatIs: (that: Any)String

We can now try out this method on a number of different inputs and observe the result:

scala> whatIs(book1)
res2: String = Scala for Java Developers is a book about Scala
scala> whatIs(book2)
res3: String = "Book(Effective Java,12.0) is a book "
scala> whatIs(books)
res4: String = List(Book(Scala for Java Developers,10.0), Book(Effective Java,12.0)) is a list of books
scala> whatIs("Scala pattern matching")
res5: String = Scala pattern matching is something about Scala
scala> whatIs("Love")
res6: String = You tell me !

Using Play JSON

There are many alternative libraries one can use to manipulate JSON in addition to the default implementation of the Scala library. In addition to the ones built on top of the known Java libraries such as Jerkson (built on top of Jackson) and other known implementations such as sjson, json4s, or Argonaut (functional programming oriented), many web frameworks have created their own including lift-json, spray-json, or play-json. Since in this book we are mostly covering the Play Framework to build web applications, we are going to focus on the play-json implementation. Note that play-json can also be run as standalone since it consists of a single jar without other dependencies to Play. Running an REPL console from within a Play project already includes the play-json dependency so that we can directly experiment with it in a console terminal window.

Note

If you want to run the following samples into an REPL different from the Play console (for instance, a regular SBT project or a Typesafe activator project) then you will have to add the following dependency to your build.sbt file:

libraryDependencies += "com.typesafe.play" %% "play-json" % "2.2.1"

scala> import play.api.libs.json._
import play.api.libs.json._ 
scala> val books = Json.parse("""
       {
         "Library": {
           "book": [
             {
               "title": "Scala for Java Developers",
               "quantity": 10
             },
             {
               "title": "Programming Scala",
               "quantity": 20
             }
           ]
         }
       }
       """)
books: play.api.libs.json.JsValue = {"Library":{"book":[{"title":"Scala for Java Developers","quantity":10},{"title":"Programming Scala","quantity":20}]}}

The JsValue type is the super type of the other JSON data types included in play-json and is listed as follows:

Programmatically, creating a JSON Abstract Syntax Tree (AST) equivalent to our list of books can therefore be expressed as follows:

scala> val books = JsObject(Seq(
         "books" -> JsArray(Seq(book1,book2))
       ))
books: play.api.libs.json.JsObject = {"books":[{"title":"Scala for Java Developers","quantity":10},{"title":"Programming in Scala","quantity":15}]}

Play has recently been enhanced to provide a slightly simpler syntax when creating the JSON structure we have just described. The alternative syntax to construct the same JSON object is given as follows:

scala> val booksAsJson = Json.obj(
         "books" -> Json.arr(
           Json.obj(
             "title" -> "Scala for Java Developers",
             "quantity" -> 10  
           ),
           Json.obj(
             "title" -> "Programming in Scala",
             "quantity" -> 15  
           )
         )
       )
booksAsJson: play.api.libs.json.JsObject = {"books":[{"title":"Scala for Java Developers","quantity":10},{"title":"Programming in Scala","quantity":15}]}

Serializing the JsObject to its string representation can be achieved with the following statement:

scala> val booksAsString = Json.stringify(booksAsJson)
booksAsString: String = {"books":[{"title":"Scala for Java Developers","quantity":10},{"title":"Programming in Scala","quantity":15}]}

Finally, since a JsObject object represents a tree structure, you can navigate within the tree by using XPath expressions to retrieve various elements, such as the following example to access the titles of both our books:

scala> val titles = booksAsJson \ "books" \\ "title"
titles: Seq[play.api.libs.json.JsValue] = ArrayBuffer("Scala for Java Developers", "Programming in Scala")

As the return type is a sequence of JsValue objects, it can be useful to convert them into Scala types and the .as[…] method would be convenient to achieve that:

scala> titles.toList.map(x=>x.as[String])
res8: List[String] = List(Scala for Java Developers, Programming in Scala)