You have a database with more than 5 000 000 000 of dots and you want to reduce it with a frozen number of lines ?
BE CAREFULL WITH THE USE OF FROZEN IDMX : It is at the origin of many data problems in the databases especially when we receive data afterwards following an interruption of sending.
You can use it only if you are receiving data in the correct chronological order.
To create a frozen number of IDMX, follow the steps below and read with attention the "What you need to know" information.
Steps to follow
1/ Create a java "frozen_IDMX" in your historical table in MX in order to attribute a new IDMX number to each line. This number resets to zero once the maximum allowed has been reached.
For example you want to have a maximum of 20 000 001 lines in your Braincube database. The java code is (with str1 = frozen_IDMX/in.java and str2 = datetime) :
if (_str2.equals("")) return "";
if (Str.ToDouble(_doJava.get("_str1",-1))< 20000001)
return Str.SetLen0(String.valueOf(Str.ToIntOrZero(_doJava.get("_str1",-1)) +1),8) ;
return Str.SetLen0("1",8);
This java has to be put in "compute everytime".
If you have more lines in this table that the maximum IDMX chosen, you will have several set of same frozen_IDMX and it's normal. The data set you want in Braincube is the most recent.
2/ Create an Output table that will contain only 20 000 001 lines (in our example) thanks to a Copy. Your source table has to be ordered by index (datetime) to have always the most recent datetime.
The resulting index of this new table is now created with frozen_IDMX/in.java, you give it the name you want, here "IDMX/in.idxr", and the IDMX of this new table is now restricted to 20 000 0001 lines
3/ Choose the Export CSV Braincube operation (change it from the default one) and fill the LF key with IDMX in Braincube database.
!!!!!! LF Key = IDMX and not the java you created !!!!!!
What you need to know
For now we have problems with CATEGORY variables that have unique modalities. Even if data is replaced by a new one at each cycle, CATEGORY dico is not cleaned and it can increase too much depending of the database frequency : gitlab.ipleanware.com
Consequence is that the database becomes unusable (it is very very slow).
A temporary action is to reimport this type of CATEGORY when you have difficulties to use a database to clean the dico.