Use of contexts within Talend

When developing jobs in Talend, it’s sometimes necessary to run them on different environments. For other business cases, you need to pass values between multiple sub-jobs in a project. To solve this kind of issues, Talend introduced the notion of “contexts”.

In this blogpost we elaborate on the usage of contexts for easily switching between a development and a production environment by storing the connection data in context variables. This allows you to determine on which environment the job should run, at runtime, without having to recompile or modify your project.

To start using contexts in Talend you have two possible scenario’s:
1) you can create a new context group and its corresponding context variables manually, or
2) you can export an existing connection as a context.
In this example we’ll go over exporting an existing Oracle connection as a context.

Double click an existing database connection to edit it and click Next. Click Export as context

Image

NOTE There are some connections that don’t allow you to export them as a context. In that case you’ll have to create the context group and its variables manually, add the group/variables to your job, and use the variables in the properties of the components of your job.

After you’ve clicked the Export as context button you’ll see the Create/Edit context group screen. Enter a name, purpose and description and click Next.

Image

Now you’ll see all the context variables that belong to this context group. Notice that Talend has already created all the context variables that are needed for the HR connection. If you want to change their names you can simply click them and they become editable.

Click the Values as table tab.

Image

In the Values as table tab you can edit the values of the context variables by simply clicking the value and changing it. To add a new context, click the context symbol in the upper right corner.

Image

The window that pops up is used to manage contexts. To create a new context, click New, enter the name of the context, in our example Production, and click Ok. To rename the Default context, select it, click Edit, enter Development and click Ok. When you’re done editing, click Ok.

Image

After the window closes, you’ll see that an extra column appeared. Enter the connection data of the production environment in the Production column and click Finish.

Image

In the connection window it’s possible to check the connection again, but this time you’ll be prompted which connection you want to check.

Image

Verify that both the connections work and click Finish.

Now that we’ve exported the connection as a context, it’s possible to use it in a job. Create a new job, use the connection that has been exported as a context and connect it to a tLogRow component. Your job should look something like this

Image

When using a connection that has been exported as a context in a job, you have to include the context variables in order for your job to be able to run. Go to the context tab and click the context button in the bottom left.

NOTE When using one of the newer versions, Talend proposes to add missing context variables whenever you try to run a job, because of this you don’t need to add them manually as described in this example.

Image

Select the context group that contains the context variables, in our case the HR context group.

Image

Select the contexts you want to include and click OK

Image

NOTE A context group can also be added to a job by simply selecting the context from the repository, dragging it towards the context tab of the job, and dropping it there.

Once you’ve added the context group to the job, it’s possible to run the job for both the development and production environment by selecting the context in the dropdown menu of the Run tab.

Image

Introduction to Websockets and JSON-P API in JEE7

Websockets (JSR 356) and the JSON-Processing API (JSR 353) are both introduced in the JEE7 specification. Together with JavaScript an HTML5, they enable web applications to deliver a richer user experience.

Websockets allow you to communicate bidirectional and full duplex over TCP, between your server and different kind of clients (browser’s, JavaFX… ). It’s basically a push technology, where, for example events or data originating from the server or a client, can be pushed to all the other connected clients.

In our demo , JSON strings are send between client and server, so that’s where the JSON Processing API comes in. It’s a  portable API that allows you to parse, generate, transform and query JSON by using the streaming or model API. But you could also send XML or any other proprietary format.

Serverside components

  1. A java class annotated with
    @ServerEndpoint(value=”/endpoint”, decoders=EncodeDecode.class, encoders=EncodeDecode.class)
    with following method annotations :
    @OnOpen : when connections is open
    @OnMessage : when a message comes in
    @OnClose : when a message is closed
  2. A java class that encode/decodes the message from/to JSON and Java object. (That’s where the JSON-P API comes in).

Clientside component

An html file that contains JavaScript to communicate with our server endpoint. Communication is done through a WebSocket object, declared as follows :

connection = new WebSocket(‘ws://localhost:8080/mywebsocket/endpoint’);

will trigger the @OnOpen method of our server side endpoint.

connection.onmessage : fired when a message comes in

connection.send : will trigger the OnMessage annotated method of our endpoint

connection.close : will trigger the OnClose annotated method of out endpoint

Demo

It’s a screen that sends messages to all the connected clients, including itself. When the client opens a connection on the server, his session is added to a list of active sessions. When a client sends a message to the server, it is distributed to all the sessions in the list. When the client closes his browser tab or window, his session is removed from the list. The data that we send, can be any complex JSON or XML model. To keep it simple, we just send a simple string.

This application needs to be deployed on a JEE7 compliant servet. So at this moment (May 2014) it will only run on Glassfish 4.0 or WildFly 8.

The war file can be found here. After deployment, open url (for Glassfish) http://localhost:8080/mywebsocket/socket.html.

 The Code

Java endpoint

package be.iadvise.mywebsocket;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import javax.websocket.EncodeException;
import javax.websocket.OnClose;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;

@ServerEndpoint(value="/endpoint", decoders=EncodeDecode.class, encoders=EncodeDecode.class)
public class MyEndPoint {
 // contains list of active sessions
 private static List<Session> sessions = Collections.synchronizedList(new ArrayList<Session>());

 @OnOpen
 public void onOpen (Session s) {
 sessions.add(s);
 System.out.println("Open session : no of sessions = "+sessions.size());
 }

 @OnMessage
 public void onMessage (MyMessage msg, Session s) throws IOException, EncodeException {
 for (Session session : sessions) { // loop over active sessions and send the message.
 session.getBasicRemote().sendObject(msg);
 }
 }
 @OnClose
 public void onClose (Session s) {
 sessions.remove(s); // remove session from the active session list.
 }
}

Java Decode/Encode message

package be.iadvise.mywebsocket;

import java.io.Reader;
import java.io.StringReader;
import java.io.StringWriter;

import javax.json.Json;
import javax.json.JsonObject;
import javax.json.JsonReader;
import javax.json.stream.JsonGenerator;
import javax.websocket.DecodeException;
import javax.websocket.Decoder;
import javax.websocket.Encoder;
import javax.websocket.EndpointConfig;

/**
 * This class will encode/decode the messages from/to the client.
 * Decoder : from client to server -> converts the JSON to MyMessage object
 * Encoder : from server to client -> converts MyMessage object to JSON
 *
 * We are using JSON, but you can use XML or any other format.
 */
public class EncodeDecode implements Decoder.Text<MyMessage>, Encoder.Text<MyMessage> { 

 @Override
 public MyMessage decode(String txt) throws DecodeException {
 Reader reader = new StringReader(txt);
 JsonReader jsonReader = Json.createReader(reader);
 JsonObject object = jsonReader.readObject();
 String text = object.getJsonString("text").getString();
 return new MyMessage (text);
 }

 //Check if decode is possible. If not, return false
 @Override
 public boolean willDecode(String s) {
 System.out.println("Will decode asked for " + s);
 return true;
 }

 @Override
 public void init(EndpointConfig config) {
 System.out.println("init called on chatdecoder");
 }

 @Override
 public void destroy() {
 System.out.println("destroy called on chatdecoder");
 }

 @Override
 public String encode(MyMessage object) {
 System.out.println("I have to encode " + object);
 StringWriter sw = new StringWriter();
 JsonGenerator generator = Json.createGenerator(sw);
 generator.writeStartObject();
 generator.write("text", ((MyMessage)object).getText());
 generator.writeEnd();
 generator.flush();
 String answer = sw.toString();
 System.out.println("I encoded an object: " + answer);
 return answer;
 }
}

Java message


package be.iadvise.mywebsocket;

public class MyMessage {
private String text;
public MyMessage(String text) {
super();
this.text = text;
}
public String getText() {
return text;
}
public void setText(String text) {
this.text = text;
}
@Override
public String toString() {
return "MyMessage [text=" + text + "]";
}
}

The html file


<html>
<head>
<script language="javascript">
var connection;
var me;
function openSocket() {
connection = new WebSocket('ws://localhost:8080/mywebsocket/endpoint');
connection.onmessage = function(evt) {
var x = JSON.parse(evt.data);
mytext = x.text;
var chld = document.createElement("p");
chld.innerHTML = mytext;
var messages = document.getElementById("messages");
messages.appendChild(chld);
}
}

function talk() {
var txt = document.getElementById("msg").value;
var message = {
'text':txt
};
connection.send(JSON.stringify(message));
}
function closeSocket() {
alert('closing socket')
connection.onclose = function () {}; // disable onclose handler first
connection.close();
}
</script>

<script type="text/javascript">
if (window.addEventListener) { // all browsers except IE before version 9
window.addEventListener ("beforeunload", closeSocket, false);
}
else {
if (window.attachEvent) { // IE before version 9
window.attachEvent ("onbeforeunload", closeSocket);
}
}
</script>
</head>
<body onLoad="openSocket();">
<p>
SimpleWebSocket
</p>
<!-- <table id="chatbox" style="display:none"> -->
<table id="chatbox">
<tr><th width="400">messages</th></tr>
<tr>
<td width="400" id="messages">
</td>
</tr>
<tr>
<td>
<input type="text" id="msg"/>
<input type="submit" value="send" onclick="talk(); return false;"></input>
</td>
</tr>
</table>
</body>
</html>

 Conclusion

Websockets are a huge improvement for building rich applications. This is the first time that push technology is actually build in the JEE framework. Before that, we had to use polling or other techniques in order to get the same results. In this blog, I showed that you don’t need much code to start off. Once you get this working, you can gradually go further building more complex sockets.