Watch out with function result cache based on data dictionary views

Result cache is a powerful tool to gain performance in PL/SQL.
There are many examples on the internet that proves this, e.g. these articles on All things Oracle:
Result Cache(1)
Result Cache(2)

But I’m not going to talk about performance.
This article is some kind of warning.

First I’ll show you how result cache works on a normal view.
I’ll create a table, a view on this table and a function that counts the rows in the view.

SQL> create table x (field1 varchar2(1), field2 number(1));

Table created.

SQL> create or replace view vie_x as select * from x;

View created.

SQL> CREATE OR REPLACE FUNCTION vie_x_rowcount(p_field1 IN VARCHAR2)
RETURN NUMBER RESULT_CACHE
IS
   l_return NUMBER;
BEGIN
   SELECT count(*)
     INTO l_return
     FROM vie_x
     WHERE field1 = p_field1;

   RETURN l_return;

END vie_x_rowcount;
/

Function created.

SQL> insert into x(field1, field2) values('x', 1);

1 row created.

SQL> commit;

Commit complete.

These are the statistics for the result cache, just to show you we’re starting without any caching.

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       0
Find Count		               0
Invalidation Count	           0

When we execute the function, the statistics show that there’s an entry created in the cache.

SQL> select vie_x_rowcount('x') from dual;

VIE_X_ROWCOUNT('X')
-------------------
		  1

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       1
Find Count		               0
Invalidation Count	           0

When we execute the same code again, we’ll get the same result and the statistics show us that the result is found in the cache.
Good job Oracle!

SQL> select vie_x_rowcount('x') from dual;

VIE_X_ROWCOUNT('X')
-------------------
		  1

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       1
Find Count		               1
Invalidation Count	           0

Let’s insert a new row in the table.
This time the statistics show us that the cache is “invalidated”, meaning the function has to be executed again to return the correct value.

SQL> insert into x values('x', 2);

1 row created.

SQL> commit;

Commit complete.

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       1
Find Count		               1
Invalidation Count	           1

And the expected result…

SQL> select vie_x_rowcount('x') from dual;

VIE_X_ROWCOUNT('X')
-------------------
		  2

The Oracle database has its own data dictionary, a set of tables where it stores all information about the database and what’s in it.
Data of these tables are available through views, data dictionary views.
In the following example I’ll use the data dictionary view that holds the information on columns.
I created a function that returns the number of columns for a certain table.

SQL> CREATE OR REPLACE FUNCTION number_of_columns(p_table_name VARCHAR2)
RETURN NUMBER RESULT_CACHE
IS

   l_return NUMBER;

BEGIN

   SELECT count(*)
     INTO l_return
     FROM user_tab_columns
    WHERE table_name = p_table_name;

   RETURN l_return;

END number_of_columns;
/

Function created.

To make sure we’ll start with a clean cache, I’ll flush it using the dbms_result_cache.flush procedure.

SQL> execute dbms_result_cache.flush

PL/SQL procedure successfully completed.

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       0
Find Count		               0
Invalidation Count	           0

When we execute the function, we’ll get the expected result: the function is executed and a cache entry is created.

SQL> select number_of_columns('X') from dual;

NUMBER_OF_COLUMNS('X')
----------------------
		     2

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       1
Find Count		               0
Invalidation Count	           0

We can execute it again and see that the return value is retrieved from the cache.

SQL> select number_of_columns('X') from dual;

NUMBER_OF_COLUMNS('X')
----------------------
		     2

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       1
Find Count		               1
Invalidation Count	           0

Let’s add a column to the table.
This should add a new row in a data dictionary table and thus in the data dictionary view we use in our function.

SQL> alter table x add (field3 date);

Table altered.

SQL> desc x
 Name					   Null?    Type
 ------------------------- -------- ----------------------------
 FIELD1 					        VARCHAR2(1)
 FIELD2 					        NUMBER(1)
 FIELD3 					        DATE

Now execute the function again.
And the result is…

SQL> select number_of_columns('X') from dual;

NUMBER_OF_COLUMNS('X')
----------------------
		     2

Not what we expected!
When we take a look at the result cache statistics, it shows that the cache wasn’t invalidated and the result was retrieved from the result cache.

SQL> select name, value from v$result_cache_statistics where name in ('Create Count Success', 'Find Count', 'Invalidation Count');

NAME			               VALUE
------------------------------ ------------------------------
Create Count Success	       1
Find Count		               2
Invalidation Count	           0

When we flush the cash and execute the function again, we’ll get the correct result cache.

SQL> execute dbms_result_cache.flush

PL/SQL procedure successfully completed.

SQL> select number_of_columns('X') from dual;

NUMBER_OF_COLUMNS('X')
----------------------
		     3

So, it seems that the result cache isn’t invalidated on data dictionary tables.
And indeed this is what I found in the Oracle documentation:

You cannot cache results when the following objects or functions are in a query:

  • Temporary tables and tables in the SYS or SYSTEM schemas

APEX 5.0: Modal dialogs have never been so easy!

Like all of us, I have had the pleasure to take a look at APEX 5.0 EA1. One of the most anticipated features was build-in support for modal dialogs. In this blog we will examine this feature more closely.

An easy way to investigate this feature is by creating two simple pages: report & form using the wizard.

0Modaldialogs

Follow the wizard and base the report & form on the demo_customer table. When you come to the page where you are creating the form then you will notice the option “Page Mode”. This mode will determine the type of page you use.

1wizard

As you can see there are three modes:
–          Normal: used by normal pages
–          Modal: when this is selected your page will show as a modal dialog
–          Non – Modal:  in this mode the page will act as a normal pop-up allowing you to continue working in both windows. Note that the original window that called the pop-up can still interact with this pop – up

We select the modal mode, and continue the wizard. We have now created two pages with one being a modal dialog. So we see here that the modal dialog is a separate page, and not some region on the page that calls it. This will allow better re-usability, and easier validations. You might notice there is no code anywhere that actually calls the modal page. That is because any page that branches, links or redirects to a page with mode “Modal”, will call that page as a modal dialog automatically.

Apart from the mode difference between the two pages, APEX also created a “Close Dialog” process, and a dynamic action that fires when the cancel button is pressed on our modal page. Both of these will make sure our modal dialog is closed when the end-user presses a button.

2Close dialog

Lastly we might want our report to refresh when the dialog is closed. To do this we go to our report page and right click on our report to create a dynamic action. As event select “Dialog Closed” and region “Customers”.

3DA event4DA action

Then right click on your Dynamic action and create a Refresh action with affected item our Customers region. Now our Customers region will be refreshed when our modal dialog closes.

Time to test our page!

5Dialog

As you can see our modal dialog renders nicely.

There is one last note I would like to add, for the developers who like to write the code for their modal dialog themselves, the APEX developers have enabled us to do so. When you go to the theme for our modal page, then in the bottom you will notice these codes:

6Dialog init code

It basically means that, should you wish to, you can modify this code here, or just replace it with your own, allowing for maximum flexibility!

In conclusion I think it is safe to say that the APEX developers have done an ACE job implementing this feature, thumbs up! If you would like to try the EA and modal dialogs for yourself then you can do that here. If you want to read up on the full list of APEX EA1 features you can do that here.

Browser and Session value in APEX

For those of us who have been developing in APEX for a while will understand that the value of an item in APEX is not always what it seems. That’s right; an item has two values in APEX. The first value is the session value that we can store in the database (server side), the second value is the value the end user sees on his screen in the browser (client side). On some occasions these values are the same. On others they are not. The difference in value can lead to some confusion, especially for the new APEX developers.

Let’s start with an example so we understand the problem. This is actually based on a real use case. I changed the item names and queries to make it more universal, but the principle remains the same. The developer in question had a page with two Select lists containing the following queries:


SELECT 1 d, 1 r
FROM dual

And

SELECT 1 d, 1 r
FROM DUAL
UNION
SELECT 2 d, 2 r
FROM DUAL

The second select list (called P55_SELECT_LIST_2), also had a default value of 2.

When the user selects a different value on the select list then the second select list had to change values accordingly.

In order to achieve this the developer had created a Dynamic Action. As event he had a change Event of the P55_SELECT_LIST item. The first action was a set value containing the following query (again made simple for the example):

SELECT 1
FROM dual
WHERE  1 = :P55_SELECT_LIST

And page items to submit he had selected the item P55_SELECT_LIST, and affected items he put P55_SELECT_LIST_2

The last action he set was a refresh action of our P55_SELECT_LIST_2 item.

Our page setup now looks like this:

1pagesettup

The developer tried his application, and when he changed the first select list he saw the second select list being refreshed. Yet, instead of displaying the correct value (1), it displayed the default value.

2items

So what went wrong?

Let’s see what exactly happened to figure that out:

  1. The page was loaded. The first select list had nothing selected, the second select list had “2” selected in the browser. Both items were empty in the session3session
  2. The value of the first select list was changed by the end user
  3. This triggered the change event defined in the dynamic action
  4. The first action “Set value” was executed. The value in the browser was changed. The session value of the item P55_SELECT_LIST  was set to one, because it was set with items to submit. An AJAX request was made to set this value. The value item P55_SELECT_LIST_2 remains NULL.4session
  5. The refresh action was executed; an AJAX request was made to get new values for the select list. Since the SELECT_LIST_2 item has no value in the session, the default value is taken, which is 2!                                                                                                                          5network

The solution was simple. Just remove the refresh action. Then the value is set in the browser correctly.  This is handled by APEX with JavaScript/jQuery. The session value of the item will remain NULL until we submit the item to the database by a page submit, a dynamic action, or by changing the value in an AJAX Callback process.

Caching in a JEE : don’t write it yourself, use LoadingCache from Google Guava libraries.

Caching data is something you use in almost every JEE project. Most of the time it’s pretty simple : put your data in a .properties file and use a PropertyManager to fetch the data.

But that’s not very flexible and manageable. Updating the values means, updating your property file, repackaging the ear file, and redeploying, and only developers can update the data.

Putting the data in JNDI entries, and using JNDI lookups may solve the problem of redeploying, but if you got a few 100 properties, it’s still not very manageable.

Most of the times, JNDI entries are entered via some application server console which, in a production environment, is not accessible for your users who need to manage this data.

So lets put the data that needs to be cached in a database, or make it accessible via a web service. That would be ideal. You can write your own application on it, and have the data managed by your users.

But that means that you have to write your own, thread safe, caching algorithms.

No big deal if the data only changes once every 10 years, but refreshing it on a time or size basis, makes the whole thing a bit more complicated. And that’s where the great LoadingCache class from the Google Guava library comes in.

What are the Guave libraries ? Well, here’s how they describe it : ‘The Guava project contains several of Google’s core libraries that we rely on in our Java-based projects: collections, caching, primitives support, concurrency libraries, common annotations, string processing, I/O, and so forth.’

Now for caching, the Guava LoadingCache class caches data in a key-object map, and lets you define a cache refreshing mechanism, all done in a thread safe manner.

So lets show a small  example and explain how it works. Suppose your cache contains a list of products that are on sale for 1 day. Depending on the no. of sold products, the price will increase during that day. This means that the cache should be updated every few seconds, to update the price, and after 1 day, the whole cache should be refreshed with new products. Suppose that price setting and product selections are in the database, updated by some back-end application, and we need the new data in our frontend application and we want to cache it.

All this can be done with this simple class :

import java.util.concurrent.TimeUnit;
import javax.ejb.EJB;
import javax.ejb.Singleton;
import be.iadvise.dao.DatabaseDAO;
import be.iadvise.entities.Product;
import com.google.common.base.Optional;
import com.google.common.cache.CacheBuilder;
import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
import com.google.common.util.concurrent.MoreExecutors;

@Singleton
public class ProductCache {

@EJB
 DatabaseDAO databaseDAO;
 private static final Integer REFRESH_PRODUCT_AFTER_5_SECONDS = 5;
 private static final Integer EXPIRE_PRODUCT_AFTER_1_DAY = 1;
 private final LoadingCache<String, Optional<Product>> cache;

 public ProductCache() {
      cache = CacheBuilder.newBuilder()
           .expireAfterWrite(EXPIRE_PRODUCT_AFTER_1_DAY, TimeUnit.DAYS)
           .refreshAfterWrite(REFRESH_PRODUCT_AFTER_5_SECONDS, TimeUnit.SECONDS)
           .build( new CacheLoader<String, Optional<Product>>() {
                 @Override
                 public Optional<Product> load( String productId ) throws Exception {
                     return loadCache(productId);
                 }
           }
     );
 }

 public Optional<Product> getEntry( String productId ) {
      return cache.getUnchecked( productId );
 }

 private Optional<Product> loadCache(String productId) {
      Product product = databaseDAO.getProduct(productId);
      return Optional.fromNullable(product);
 }
}

Explanation

  1. In the constructor, we build the cache using the CacheLoader, defining the refresh mechanism. In our example we define 2 rules :
    – expireAfterWrite : after this period, the object will be evicted from the cache, and replaced the next time it is requested.
    – refreshAfterWrite : after this period, the object will be refreshed using the loadCache method. (with our new price)
  2. getEntry(String productId) method : will return the object with given key. So in this example, the cache is not loaded all at once, but only when the object is needed.
  3. loadCache(String productId) : will load the product and add it to the cache, or replace it if it’s already there and needs to be refreshed.

That’s all there is to it !

A few other remarks on the code

  1. There are other mechanism like expire/refresh AfterRead, which will time only from the last read, or let the cache hold only a certain no. of objects,…
  2. This code is implemented as a session bean. To make a singleton, I’m using EJB 3 annotation @Singleton. Because I only want 1 cache in my application
  3. My DAO is also injected using the @EJB annotation
  4. The LoadingCache does not want any null objects in the map (returns an error), so I’m using the Guava ‘Optional’ class here. This is basically a wrapper for my object and used to check if there is a value for my product id or not. So if someone uses a wrong productId, my cache will indicate that there is no product for this id, and I don’t have to go to the database every time it is requested.

To conclude:

Programming a caching mechanism in a JEE environment is not as trivial as it may seem. Testing it in a multithreaded environment is even more difficult. The caching classes of Guave gives you ready-to-use solution. It’s programmed, tested and used by Google, so I think we can say in all honesty : this is proven technology.

A remark on deploying on Weblogic 12c:

Weblogic also uses the Guava libraries, but an older version. This causes following error on deployment :

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()

Lcom/google/common/util/concurrent/ListeningExecutorService;

Adding the following to your weblogic-application.xml will solve the problem (force weblogic to use your deployed Guava libraries :

<wls:prefer-application-packages>
<wls:package-name>
com.google.common.*
</wls:package-name>
<wls:prefer-application-packages>

Guava libraries run under Apache license, more info/download can be found on :

https://code.google.com/p/guava-libraries/

Have fun !

AJAX in APEX

AJAX is becoming important in the world of web applications. APEX has provided us a very easy way to create an AJAX process, by using dynamic actions. Using PL/SQL Actions in Dynamic Actions to communicate with the database without submitting the page will suffice in most cases, but the downside is that the code is not very re-usable, and when you want to write a plug-in you simply don’t have access to Dynamic Actions. In this blog you will learn how to code your own AJAX process.

An AJAX process in APEX consists out of three parts

  • The JavaScript code that calls the AJAX PL/SQL Process
  • The PL/SQL Process that might or might not return a value
  • The JavaScript code that catches the return value and possibly does something with it

In APEX there are three ways to create an AJAX process from JavaScript:

  • The htmldb_get() method: undocumented but this used to be the only method available (without installing external libraries)
  • jQuery.ajax(): since jQuery was added to APEX, it has been quite common to use this method. It’s well documented on the jQuery homepage, but the downside is you need to write more code
  • apex.server: this new APEX API has been recently added (I believe at APEX 4.2). It is actually a wrapper of jQuery.ajax(), so it supports the same functionality with some additional APEX specific features. It is thoroughly documented in the APEX documentation, and this is the reason I prefer this method, and I will explain how you too can use it

The first thing we do is create a test application. In our case we have a table called “JOBS” that looks like this:

jobstable

In my jobs table I just inserted one job with a salary of 2800 of an unknown currency.

In our APEX application we have an Item of the type select list where the user can select a job, and then the minimum salary will be filled in.

Our page looks like this:

page

Next we write our JavaScript code.  This includes our change event and the apex.server.process . Double click your page name to go to the page definition, and scroll down to “Execute when page loads”.

javascript call

  • AJAX_GET_MIN_SALARY is the name of our future AJAX process.
  • X01 is the variable we pass, in this case the value of our #P17_JOB_ID item
  • Finally we declare that our expected return type is plain text. If we don’t do this, then by default the function expects a JSON string returned. Furthermore we declare in this function what we do with this return data. The return data will be delivered asynchronous, meaning we will get this data from our AJAX Callback function as soon as the AJAX Callback process is ready.

Now we can create our AJAX_GET_MIN_SALARY Ajax Callback process. Just right click on Ajax Callbacks . Click “Create” and select PL/SQL. Here we can put our PL/SQL code:

ajax_process

There are two things here that are worth mentioning:

  • TO_CHAR(apex_application.g_x01): this is how we catch the variable that is passed from our JavaScript code. We use TO_CHAR to identify that it’s a character.
  • HTP.Prn(v_min_salary): here we return the minimum salary back to our page

There, all done!  Let’s test out our application, shall we? Before you do anything it’s best to open the developer toolbar in the browser. In Chrome you can do this by pressing ctrl+shift+J.  It’s  a good practice to reload the page and to check if any JavaScript errors pop up on the console. If our JavaScript code shows no errors in the console go to the ‘Network’ tab, and select a job in the application.

items

You will now see www_flow.show appear. Click it. There are two tabs here that are vital to investigating this function for debugging, if needed. The first is the header, it shows what data is send to our AJAX Callback function.

toolbar

The second tab that’s important is our Response tab. It tells us what data is send back from the PL/SQL Process. If you remember our PL/SQL Process you will notice that we did not include an exception for when no data was found. Select “null” as job and you will get an error. If you then check out the response of the AJAX call you will see it gives our ORA error.

error

If you managed to read this far then you have gained some insights on how you can create your own AJAX function using the new APEX JavaScript API, how it works and how you can debug it should not everything go as planned.

Migrate your MS Access data to an Oracle database using the ETL Tool Talend

APEX is promoted as the perfect replacement for MS Access applications. One thing you should consider though is how you migrate your data to the Oracle database. In APEX there is a handy tool called the Data Workshop that can be used for this. You first export your Excel files from the MS Access database, and then follow the data upload wizard to import the data into identical tables. Since you are not always working with a 1-1 relationship, you will most likely have to write some PL/SQL to get all the data in the right tables.

dataworkshop

The downside is that you will need to repeat this process when you go into production. This is not a big problem if you only have one table to migrate. But if you have multiple tables and/ or your users also want new data during tests and trainings, you will spend a lot of time exporting and importing Excel files.

A recent APEX project for a client required a large data migration from MS Access Databases to the Oracle database. Because we would require fresh data on several points in the development process we decided to use the ETL Open Source Tool Talend. We got impressed of how intuitive the tool is, it only took a few days before we were familiar with the tool. Once you get the hang of it, you can write (or should I say draw) migrations of tables in no time. We needed to migrate from an MS Access database but the tool supports a wide range of databases and documents to import your data from. In total we migrated around 30-40 tables to our Oracle database.

Let’s have a closer look at one of our migration jobs.

talendmsaccessoracle

At the left we see our MS Access database. Each tAccessInput component will get data from one table. After that we join the tables in our tMap_1 component. The reason we don’t just write our joins in one component, is because this way we can really see how many rows every table returns.

On the bottom we have some Oracle Database input connections. They will join the persons of our MS Access Database with the persons in our Oracle Database based on the National registration number. After that we write our data to our Oracle Database. You may notice that we have two lines going to Excel files. This is our error logging; we use this to log the rows that did not find a match. In our first Excel for example we write persons that did not find a match in our Oracle Database.

This is just one example, in total about 20 jobs were built. During the development we also had to deal with certain calculations or convert data. For most things there was a component ready to use and if there wasn’t you could always write a Java expression in the tMap items.

I hope I convinced you of the benefits of using Talend as a migration tool for APEX projects, because we will certainly use this tool again!

Connecting to Salesforce and Mailchimp using Talend

A lot of companies use Salesforce to manage their customers and contacts. In addition Mailchimp can be used for sending out mailings to these connections. Mailchimp also captures information about what people did with these mails. This can be useful information for your CRM. A while ago, I was asked to make a list of everyone that have opened their mails in Mailchimp. Let me show you how easy it is, to do something like that with Talend.

In Talend:

  • we can get a list of email addresses from Mailchimp of receivers that opened a mail
  • and we can ask Salesforce for the email addresses and names of all our connections
  • and we can also use a mapping component to join these lists.

Talend has a standard interface with Salesforce. And Mailchimp offers lots of RESTful web services, which we can make use of in our Talend job.

  1. Connecting to Salesforce  

Right click “Salesforce” under the Metadata and choose “Create Salesforce Connection”.

pic1

After choosing a name for our connection, all we need to fill in, is the username and password for our Salesforce-connection.  The rest is already filled in for us.

pic2

To enable the “Finish” button, we need to check our properties first, using the button “Check login”.

Under Metadata, we can now browse through all our Salesforce-data.

pic3

Now you’re probably wondering, how to use this data in your ETL-flow. Well.. that’s even easier!

Simply drag one of the tables (with the blue icons) into your job and choose for the “tSalesforceInput” component from it’s 3 suggestions.

pic4

After specifying the necessary mappings you should get something like this:

pic5

We’ve used Contact and Account data of Salesforce for this.

In the next part, let’s check out how we generated the list of email addresses.

2.       Connecting to Mailchimp

Accessing your Mailchimp-data, is a bit harder. We need two components from the Talend-palette:

The ‘tRest’ component,  because we need to use a RESTful webservice for requesting our data from Mailchimp. And the ‘tExtractJSONFields’ component for interpreting the data we receive back.

After dragging the tRest component to your job, choose ‘POST’ as the ‘method’ and fill in the URL, corresponding to the report you wish to receive.

pic6

If you want to receive your report in XML-format instead of JSON, just add “.xml” at the end of the URL.

Here we needed the Mailchimp report, that gives us information on opened emails.

If you are interested in other kinds of reports, you can find the list here:

http://apidocs.mailchimp.com/api/2.0/#lists-methods

Every request, needs certain parameters. We can specify them in the HTTP body field, like this:

“{\”apikey\”: \”your api key will be here\”,\”cid\”: \”put a campaign id here\”}”

The API-key will always be needed as the first parameter. You can find it in Mailchimp under your ‘Account Settings’  – ‘Extras’ .

pic7pic8

The second component we need, is called ‘ExtractJSONFields’. After dragging it to our job, we link our first component to it.

pic9

We can use ‘Edit schema’, to define the data we want to extract.

pic10

Finally all we need to do, is specify the location of this data we are interested in, for example the ‘email’-field inside the ‘member’-field.

pic11

Now that we’re able to access our data from Mailchimp, let’s take a look at how we used it for generating the list of e-mailaddresses.

First we asked Mailchimp for all our Campaigns, then we used the ‘flowToIterate’-component so we could ask Mailchimp for the email addresses, once for every campaign in the list:

pic12

Finally all we had to do, is put these two jobs together and press ‘run’.

So.. I hope you’ll enjoy it, as much as I did!