LEI'S DESIGN SERVICES

EXPERIENCED IN CHIP DESIGN VERIFICATION

Chapter 4:  MapReduce Data Analysis Using Java

 Figure 2  Winning Number Data Analysis MapReduce Diagram

 

4.1 Dataset

a)      Input for lottery prizes:

/input/megamillionprizesR.txt

 

Code content:

1   mega

2   1Nmega

6   2Nmega

9   3

54   3Nmega

718   4

5004   4Nmega

373934   5

84000000   5Nmega

 

     b)   Lottery winning numbers input data file for each period for data analysis:

/input/Testingperiod.txt

where Testingperiods are 2015to2006, 2015to2007, 2015to2008, 2015to2009, 2015to2010, 2015to2011, 2015to2012, 2015to2013, 2015to2014, and 2015to2015.

 

             Top 6 winning Numbers input test files to analyze the input data sampling file:           

             /output/Samplingperiod_top6/part-r-00000,

            /output/Samplingperiod_ top5Nmega/part-r-00000,

            /output/Samplingperiod_top5numsNmega.txt, and

            /output/Samplingperiod_individualtop5Nmega.txt;

 

            Top 6 winning Numbers input test files in reverse order:

           /output/Samplingperiod_top6R/part-r-00000,

            /output/Samplingperiod_ top5NmegaR/part-r-00000,

            /output/Samplingperiod_top5numsNmegaR.txt, and

            /output/Samplingperiod_individualtop5NmegaR.txt;

 

            Top 6 individual winning numbers input test files left shifted from 1 to 4 numbers:

            /input/Samplingperiod _individualtop5Nmega2sl1.txt,

            /input/Samplingperiod _individualtop5Nmega2sl2.txt,

            /input/Samplingperiod _individualtop5Nmega2sl3.txt, and

            /input/Samplingperiod _individualtop5Nmega2sl4.txt;

 

           Top 6 average, median, bottom, and individual winning numbers:

           /input/Samplingperiod _individualbottomall6.txt,

           /input/Samplingperiod _all6med.txt,

           /input/Samplingperiod _all6avg.txt, and

           /input/Samplingperiod _bottomall6.txt;

           

           Top 6 average, median, bottom, and individual winning numbers in reverse order:

           /input/Samplingperiod _individualbottomall6R.txt,

           /input/Samplingperiod_all6medR.txt,

           /input/Samplingperiod _all6avgR.txt, and

           /input/Samplingperiod _bottomall6R.txt;

          where Samplingperiods are 2005to2005, 2006to2005, 2007to2005, 2008to2005, 2009to2005,

          2010to2005, 2011to2005, 2012to2005, 2013to2005, and 2014to2005.

  *** Please note Hadoop takes input file from the bottom order to the top order like a FILO, the data first in is last out to be use.  Therefore, reverse order data files is right order, and  the right order data file is the reverse order when doing Hadoop jar file execution.

       Analysis types are the methods used to do the analysis.  They are listed in the text file, megamillionAnalysisType.txt, as follow below:

 

 

top6  1

top5Nmega   2

top5numsNmega       3

individualtop5Nmega 4

top62            5

top5Nmega2 6

top5numsNmega2     7

individualtop5Nmega2           8

top63            9

top5Nmega3 10

top5numsNmega3     11

individualtop5Nmega3           12

top6R            13

top5NmegaR 14

top5numsNmegaR     15

individualtop5NmegaR           16

top62R          17

top5Nmega2R           18

top5numsNmega2R   19

individualtop5Nmega2R         20

top63R          21

top5Nmega3R           22

top5numsNmega3R   23

individualtop5Nmega3R         24

individualtop5NmegaSL1       25

individualtop5NmegaSL2       26

individualtop5NmegaSL3       27

individualtop5NmegaSL4       28

all6med         29

all6avg           30

bottomall6     31

individualbottomall6   32

all6medR       33

all6avgR        34

bottomall6R   35

individualbottomall6R 36

4.2 Problem Statement

      Do data analysis of matching winning numbers based on the following conditions:

a)      any 6 winning numbers

b)     any 5 winning numbers

c)      any 4 winning numbers and the mega number

d)     any 4 winning numbers

e)      any 3 winning numbers and mega number

f)       any 3 winning numbers

g)      any 2 winning numbers and mega number

h)      any 2 winning numbers

i)        any 1 winning number and mega number

j)       any 1 winning number

k)     only mega number

l)        no winning number

 

4.3 Approach

Use the realistic top 6 input winning numbers described in the testing period files in Section 4.1 against sampling period files for comparison to see how many conditions described in Section 4.2  are met. Sampling and testing periods go hand in hand to be realistic.  Sampling period started just after the testing period. 

For example, to test the testing period between 2015 and 2006, the sampling period is needed from beginning of 2005 to the end of 2005 for realistic data analysis.  The files for data analysis are /input/megamillionprizesR.txt, /input/2015to2006.txt, /output/2005to2005_top6/part-r-00000,

            /output/2005to2005_ top5Nmega/part-r-00000,

            /output/2005to2005_top5numsNmega.txt, and

            /output/2005to2005_individualtop5Nmega.txt.

The Java code to do the data analysis has 3 mappers, 1 reducer, and 1 main class as shown in Figure 2 above.  The main class is winningNumAnalysis.java.  The 3 mapper classes are mostFreqNumMapper.java, winningPrizesMapper.java, and winningNumAnalysisMapper.java, where one mapper parses in top 6 input test data from the sampling period, another mapper parses in winning Mega Millions prizes data, and finally the testing period Mega Millions data. The reducer class is winningNumAnalysisReducer.java.

 

 

4.4 Code

The java files to run the data analysis are listed below:

Java main class file name:  winningNumAnalysis.java

 

Java code:

package com.analysis;

 

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.conf.Configured;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

import org.apache.hadoop.util.Tool;

import org.apache.hadoop.util.ToolRunner;

 

         //Driver code for winning number analysis

         public class winningNumAnalysis  extends Configured implements Tool{

          

                     public int run(String[] args) throws Exception

                     {

                        Configuration conf = new Configuration();

                        String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

                       

                        if (args.length != 4) {

                                    System.err.println("Usage:  hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis <input path1> <input path2> <input path3> <output path>");

                                    System.err.printf("Argument length: %d", args.length);   

                                   

                                    System.exit(-1);

                        }

                       

                        Job job = new Job(conf, "winningNumberAnalysis");

                       

                        job.setJarByClass(winningNumAnalysis.class);

                    

                        job.setReducerClass(winningNumAnalysisReducer.class);

                       

                        job.setMapOutputKeyClass(Text.class);

                        job.setMapOutputValueClass(Text.class);

                       

                        job.setOutputKeyClass(Text.class);

                        job.setOutputValueClass(Text.class);      

 

                           MultipleInputs.addInputPath(job,new Path(args[0]),TextInputFormat.class, winningNumAnalysisMapper.class);

                           MultipleInputs.addInputPath(job,new Path(args[1]),TextInputFormat.class, mostFreqNumMapper.class);

           MultipleInputs.addInputPath(job,new Path(args[2]),TextInputFormat.cla

ss, winningPrizesMapper.

class);

                           FileOutputFormat.setOutputPath(job,  new Path(otherArgs[3]));

                          

                           System.exit(job.waitForCompletion(true) ? 0 : 1);

                           boolean success = job.waitForCompletion(true);

                           return success ? 0 : 1;

                          

               }

                public static void main(String[] args) throws Exception {

                    winningNumAnalysis driver = new winningNumAnalysis();

                    int exitCode = ToolRunner.run(driver, args);

                    System.exit(exitCode);   

                    }

            }

         

 

Java mapper 1 file name:  mostFreqNumMapper.java

Code content:

package com.analysis;

 

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

 

import java.io.IOException;

 

public class mostFreqNumMapper extends Mapper<LongWritable, Text, Text, Text> {

               private Text num = new Text();

               public void map(LongWritable key, Text value, Context context) throws

                  IOException, InterruptedException {

                           String line = value.toString();

 

                           String[] array = line.split("\t");

                           num = new Text(array[0]);    

                           context.write(new Text("ComparisonNumber"),  num); //new IntWritable(1));

 

               }

}  //end of mostFreqNumMapper class

 

Java mapper 2 file name:  winningPrizesMapper.java

Code content:

package com.analysis;

 

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

 

import java.io.IOException;

 

public class winningPrizesMapper extends Mapper<LongWritable, Text, Text, Text>

{

            private Text num = new Text();

            public void map(LongWritable key, Text value, Context context) throws

               IOException, InterruptedException {

                        String line = value.toString();

 

                        String[] array = line.split(" ");

                        num = new Text(array[0]);    

                        context.write(new Text("WinningPrize"),  num);

            }

}  //end of winningPrizesMapper class

 

Java mapper 3 file name:  winningNumAnalysisMapper.java

Code content:

package com.analysis;

 

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

import java.util.StringTokenizer;

 

public class winningNumAnalysisMapper extends Mapper<LongWritable, Text, Text, Text> {

               private Text num = new Text();

               private String num1 = new String();

               private String num2 = new String();

               private String num3 = new String();

               private String num4 = new String();

               private String num5 = new String();

               private String meganum = new String();

               private Text num6 = new Text();

              

              public void map(LongWritable key, Text value, Context context) throws

                  IOException, InterruptedException {

                           String line = value.toString();

                           StringTokenizer tokenizer = new StringTokenizer(line, " ");

                           int i = 0; //initialize the counter

                          

                           while (tokenizer.hasMoreTokens()) {

                                                  

                                          num.set(tokenizer.nextToken());

 

                                                              if ((i >= 5) && (i <= 10))  {

 

//all 6 winning numbers                                              

                                                               if (i == 5) {//first winning number

                                                                          num1 = num.toString();  

                                                               }

                                                               if (i == 6) {//second winning number

                                                                           num2 = num.toString();

                                                               }

                                                               if (i == 7) {//third winning number

                                                                           num3 = num.toString();

                                                               }

                                                               if (i == 8) {//fourth winning number

                                                                           num4 = num.toString();

                                                               }

                                                               if (i == 9) {//fifth winning number

                                                                           num5 = num.toString();

                                                               }

                                                               if (i == 10) {//mega winning number

                                                                           meganum = num.toString();

                                                                           num6 = new Text(num1 + "\t" +

 num2 + "\t" + num3 + "\t" + num4 + "\t" + num5 + "\t" + meganum);

                                                                           context.write(new Text("Winni

ngnumbers"),  num6); //new IntWritable(1));

                                                                          

                                                               }

 

                                                               }                                          

                                      if (i < 11) { 

                                           i = i + 1;

                                          

                                                  

                                      }

 

 

                                      else

                                             i = 0; //reset counter to 0              

                                

                        }

            }

}  //end of winningNumAnalysisMapper class

 

 

Java reducer file name:  winningNumAnalysisReducer.java

Code content: 

 

 

package com.analysis;

 

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

 

public class winningNumAnalysisReducer extends Reducer<Text, Text, Text, Text> {

  //  public void reduce(Text key, Iterable<IntWritable> values, Context context)

    String line = null;

    String[] compareNum = new String[6];

    String[] winningNum = new String[100000];

    String[] WinningPrize = new String[9];

    String[] data = new String[6];

   

    int i = 0;

    int j = 0;

    int k = 0;

    int m = 0;

    int n = 0;      

    int winNumCount = 0;

//5 winning number comparison count against 6 comparison numbers

    int megaCount = 0; //mega number comparison count against 6 comparison numbers

    int win6nums = 0; //any 6 numbers

    int win5nums = 0; //any 5 numbers

    int win4numsNmega = 0; //any 4 numbers and 1 mega number

    int win4nums = 0; //any 4 numbers

    int win3numsNmega = 0; //any 3 numbers and 1 mega number

    int win3nums = 0; //any 3 numbers

    int win2numsNmega = 0; //any 2 numbers and 1 mega number

    int win2nums = 0; //any 2 numbers

    int win1numNmega = 0; //any 1 number and 1 mega number

    int win1num = 0; //any 1 number

 

    int winmeganum = 0; //1 mega number

    int winNonum = 0; //no number

 

    int profit = 0; //profit or loss

    int statTotal = 0;

    int statwin6nums = 0;

    int statwin5nums = 0;

    int statwin4numsNmega = 0;

    int statwin4nums = 0;

    int statwin3numsNmega = 0;

    int statwin3nums = 0;

    int statwin2numsNmega = 0;

    int statwin2nums = 0;

    int statwin1numNmega = 0;

    int statwin1num = 0;

    int statwinmeganum = 0;

    int statwinNonum = 0;

    int winpercent = 0;

 

           public void reduce(Text key, Iterable<Text> values, Context context)        

      throws IOException, InterruptedException {                    

 

                            for (Text text : values) {

                                line = text.toString();

                                context.write(key, new Text(line));

                                if (key.equals(new Text("ComparisonNumber"))) {

                                                     compareNum[i] = line;

                                                     i++;

                                }

                                else if (key.equals(new Text("WinningPrize"))) {

                                         WinningPrize[k] = line;

                                                     k++;

                                }

 

                                else {

                                                     winningNum[j] = line;

                                                     j++;                                   

                                }                                       

 } // end of for loop

 

         for (m = 0; m < j; m++) {

                    

                  data = winningNum[m].split("\t");

       

        //before comparison initialize the winning number count and mega number count

        winNumCount = 0;

        megaCount = 0;

       

             for (k = 0; k < 6; k++) {

                         for (n = 0; n < 6; n++) {

                                  //System.out.printf("printout WinningNumbers[%d]: %s.\n", n, data[n]);

                                  if ((compareNum[k].equals(data[n])) && (k < 5) && (n < 5

)) { //if comparison number equals winning number 1 to 5, except mega number

                                           winNumCount++;

                                             

                                  }

                                  else if ((compareNum[k].equals(data[n])) && (k == 5) &&

(n == 5)) { //if comparison number equals mega number

                                           megaCount++;

                                             

                                  }

                         } //end of for n loop

             } //end of for k loop

            

             //5 winning numbers and 1 mega number

             if ((winNumCount == 5) && (megaCount == 1)) {

                      win6nums++; //any 6 numbers

                       context.write(new Text("Won the lottery!  "), new Text(winningNum[m]));

                    

             }

             //any 5 winning numbers

             else if ((winNumCount == 5) && (megaCount == 0)) {            

                win5nums++; //any 5 numbers

                          context.write(new Text("Won 5 numbers!  "), new

 

Text(winningNum[m]));                            

             }

                //any 4 winning numbers and 1 mega number

 

          else if ((winNumCount == 4) && (megaCount == 1)) {

              win4numsNmega++; //any 4 numbers and 1 mega number

                            context.write(new Text("Won 4 numbers and 1 mega number!  "), new Text(winningNum[m]));

                 

            }

          //any 4 winning numbers

          else if ((winNumCount == 4) && (megaCount == 0)) {

             win4nums++; //any 4 numbers

                            context.write(new Text("Won 4 numbers!  "), new Text(winningNum[m]));

                 

            }

                //any 3 winning numbers and 1 mega number

 

          else if ((winNumCount == 3) && (megaCount == 1)) {

             win3numsNmega++; //any 3 numbers and 1 mega number

                    context.write(new Text("Won 3 numbers and 1 mega number!  "),

new Text(winningNum[m]));

            }

          //any 3 winning numbers

          else if ((winNumCount == 3) && (megaCount == 0)) {

             win3nums++; //any 3 numbers

                            context.write(new Text("Won 3 numbers!  "), new Text(winningNum[m]));

            }

          //any 2 winning numbers and 1 mega number

          else if ((winNumCount == 2) && (megaCount == 1)) {

             win2numsNmega++; //any 2 numbers and 1 mega number

            }

          //any 2 winning numbers

          else if ((winNumCount == 2) && (megaCount == 0)) {

             win2nums++; //any 2 numbers

            }

          //any 1 winning number and 1 mega number

          else if ((winNumCount == 1) && (megaCount == 1)) {

             win1numNmega++; //any 1 number and 1 mega number

                            context.write(new Text("Won 1 number and 1 mega number!  "), new Text(winningNum[m]));

                 

            }

          //any 1 winning number

                else if ((winNumCount == 1) && (megaCount == 0)) {

 

                win1num++; //any 1 number

                      //System.out.printf("Won 1 lottery number!\n");

                  }

             //only mega number

             else if ((winNumCount == 0) && (megaCount == 1)) {

                winmeganum++; //1 mega number

context.write(new Text("Won 1 mega number!  "), new Text(winningNum[m]));

                    

                  }

             //no winning number

             else {

                 winNonum++; //no number

             }

                    

         } // end of for m loop

        

  if (m != 0) { //don't print when m loop count is zero

System.out.printf("                       Total Summary: \n");

System.out.printf("______________________________________________________________________\n");

System.out.printf("   Jackpot(all6numbers):  %d,                    Winning Prize: $%d\n", win6nums, Integer.parseInt(WinningPrize[0])*win6nums);

System.out.printf("   any 5 winning numbers:  %d,                   Winning Prize: $%d\n", win5nums, Integer.parseInt(WinningPrize[1])*win5nums);

System.out.printf("   any 4 winning numbers and 1 mega number:  %d, Winning Prize: $%d\n", win4numsNmega, Integer.parseInt(WinningPrize[2])*win4numsNmega);

System.out.printf("   any 4 winning numbers:  %d,                   Winning Prize: $%d\n", win4nums, Integer.parseInt(WinningPrize[3])*win4nums);

System.out.printf("   any 3 winning numbers and 1 mega number:  %d, Winning Prize: $%d\n", win3numsNmega, Integer.parseInt(WinningPrize[4])*win3numsNmega);

System.out.printf("   any 3 winning numbers:  %d,                   Winning Prize: $%d\n", win3nums

 

Integer.parseInt(WinningPrize[5])*win3nums);

System.out.printf("   any 2 winning numbers and 1 mega number:  %d, Winning Prize: $%d\n", win2numsNmega, Integer.parseInt(WinningPrize[6])*win2numsNmega);

System.out.printf("   any 2 winning numbers:  %d,                   Winning Prize: $%d\n", win2nums, 0);

System.out.printf("   any 1 winning number and 1 mega number:  %d,  Winning Prize: $%d\n", win1numNmega, Integer.parseInt(WinningPrize[7])*win1numNmega);

System.out.printf("   any 1 winning number:  %d,                    Winning Prize: $%d\n", win1num, 0);

            System.out.printf("   1 mega number:  %d,                           Winning Prize:

 

$%d\n", winmeganum, Integer.parseInt(WinningPrize[8])*winmeganum);

   System.out.printf("   no number:  %d,                               Winning Prize: $0\n", winNonum);

 

   profit = -win6nums + Integer.parseInt(WinningPrize[0])*win6nums - win5nums + Integer.parseInt(WinningPrize[1])*win5nums

      - win4numsNmega + Integer.parseInt(WinningPrize[2])*win4numsNmega - win4nums

              + Integer.parseInt(WinningPrize[3])*win4nums - win3numsNmega

+ Integer.parseInt(WinningPrize[4])*win3numsNmega

      - win3nums + Integer.parseInt(WinningPrize[5])*win3nums - win2numsNmega

      + Integer.parseInt(WinningPrize[6])*win2numsNmega

              -win2nums -win1numNmega + Integer.parseInt(WinningPrize[7])*win1numNmega - win1num

              - winmeganum + Integer.parseInt(WinningPrize[8])*winmeganum - winNonum;

   System.out.printf("If spend a dollar each week to buy a lottery ticket, profit or loss:  %d\n", profit);

 

   statTotal = win6nums + win5nums + win4numsNmega + win4nums

             + win3numsNmega + win3nums + win2numsNmega +

            win2nums + win1numNmega + win1num + winmeganum + winNonum;

     statwin6nums = ((win6nums * 1000) / statTotal + 5) /10;

     statwin5nums = ((win5nums * 1000) / statTotal + 5) /10;

     statwin4numsNmega = ((win4numsNmega * 100) / statTotal + 5) /10;

     statwin4nums = ((win4nums * 1000) / statTotal + 5) /10;

     statwin3numsNmega = ((win3numsNmega * 100) / statTotal + 5) /10;

     statwin3nums = ((win3nums * 1000) / statTotal + 5) /10;

     statwin2numsNmega = ((win2numsNmega * 1000) / statTotal + 5) /10;

     statwin2nums = ((win2nums * 1000) / statTotal + 5) /10;

     statwin1numNmega = ((win1numNmega * 1000) / statTotal + 5) /10;

     statwin1num = ((win1num * 1000) / statTotal + 5) /10;

     statwinmeganum = ((winmeganum * 1000) / statTotal + 5) /10;

     statwinNonum = ((winNonum * 1000) / statTotal + 5) /10;

 

      winpercent = statwin6nums + statwin5nums + statwin4numsNmega +

                 statwin4nums + 

                  statwin3numsNmega + statwin3nums + statwin2numsNmega +

                  statwin1numNmega + statwinmeganum;

 

     context.write(new Text("   "), new Text("                       Total

Summary: "));

 

        context.write(new Text("   "), new Text("_____________________________

_________________________________________"));

  context.write(new Text("Any 6 numbers:                    \t"), new Text(Integ

er.toString(win6nums) + " is "+ Integer.toString(statwin6nums) + "% of total # of tickets sold")); //any 6 numbers

  context.write(new Text("Any 5 numbers:                    \t"), new Text(Integer.toString(win5nums) + " is "+ Integer.toString(statwin5nums) + "% of total # of tickets sold")); //any 5 numbers

  context.write(new Text("Any 4 numbers and 1 mega number:  \t"), new Text(Integer.toString(win4numsNmega) + " is "+ Integer.toString(statwin4numsNmega) + "% of total # of tickets sold")); //any 4 numbers and 1 mega number

 

  context.write(new Text("Any 4 numbers:                    \t"), new Text(Integer.toString(win4nums) + " is "+ Integer.toString(statwin4nums) + "% of total # of tickets sold")); //any 4 numbers

  context.write(new Text("Any 3 numbers and 1 mega number:  \t"), new Text(Integer.toString(win3numsNmega) + " is "+ Integer.toString(statwin3numsNmega) + "% of total # of tickets sold")); //any 3 numbers and 1 mega number

  context.write(new Text("Any 3 numbers:                    \t"), new Text(Integ

er.toString(win3nums) + " is "+ Integer.toString(statwin3nums) + "% of total # of tickets sold")); //any 3 numbers

  context.write(new Text("Any 2 numbers and 1 mega number:  \t"), new Text(Integer.toString(win2numsNmega) + " is "+ Integer.toString(statwin2numsNmega) + "% of total # of tickets sold")); //any 2 numbers and 1 mega number

  context.write(new Text("Any 2 numbers:                    \t"), new Text(Integer.toString(win2nums) + " is "+ Integer.toString(statwin2nums) + "% of total # of tickets sold")); //any 2 numbers

  context.write(new Text("Any 1 number and 1 mega number:   \t"), new Text(Integer.toString(win1numNmega) + " is "+ Integer.toString(statwin1numNmega) + "% of total # of tickets sold")); //any 1 number and 1 mega number

  context.write(new Text("Any 1 number:                     \t"), new Text(Integer.toString(win1num) + " is "+ Integer.toString(statwin1num) + "% of total # of

tickets sold")); //any 1 number

  context.write(new Text("1 mega number:                    \t"), new Text(Integer.toString(winmeganum) + " is "+ Integer.toString(statwinmeganum) + "% of total # of tickets sold")); //1 mega number

  context.write(new Text("No number:                        \t"), new Text(Integ

er.toString(winNonum) + " is "+ Integer.toString(statwinNonum) + "% of total # of tickets sold")); //no number

  context.write(new Text("Profit or loss:                   \t"), new Text("$" + Integer.toString(profit))); //profit or loss

 

context.write(new Text("Winning Percentage:               \t"), new Text(Integer.toString(winpercent) + "%")); //winning percentage 

 } //end of if statement

 

  }

 

} //end of winningNumAnalysisReducer class

 

For example, the linux script to run the data analysis for the testing period between 2015 and 2006 is listed below:

 

a)   For testing with top 6 winning numbers selected from sampling period data

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /output/2005to2005_top5Nmega/part-r-00000 /input/megamillionprizesR.txt /output/2005to2005_top5NmegaAnalysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /output/2005to2005_top5numsNmega.txt /input/megamillionprizesR.txt /output/2005to2005_top5numsNmegaAnalysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /output/2005to2005_individualtop5Nmega.txt /input/megamillionprizesR.txt /output/2005to2005_individualtop5NmegaAnalysis

 

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /output/2005to2005_top6R.txt /input/megamillionprizesR.txt /output/2005to2005_top6Ranalysis

 

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /output/2005to2005_top5NmegaR.txt /input/megamillionprizesR.txt /output/2005to2005_top5NmegaRAnalysis

 

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /output/2005to2005_top5numsNmegaR.txt /input/megamillionprizesR.txt /output/2005to2005_top5numsNmegaRAnalysis

 

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /output/2005to2005_individualtop5NmegaR.txt /input/megamillionprizesR.txt /output/2005to2005_individualtop5NmegaRAnalysis

 

b)  For testing with individual top 6 numbers shifted to left 1 to 4 positions

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_individualtop5Nmega2sl1.txt /input/megamillionprizesR.txt /output/2005to2005_individualtop5Nmega2SL1Analysis

 

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_individualtop5Nmega2sl2.txt /input/megamillionprizesR.txt /output/2005to2005_individualtop5Nmega2SL2Analysis

 

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_individualtop5Nmega2sl3.txt /input/megamillionprizesR.txt /output/2005to2005_individualtop5Nmega2SL3Analysis

 

hadoop jar /home/notroot/lab/workspacehadoopHW/winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_individualtop5Nmega2sl4.txt /input/megamillionprizesR.txt /output/2005to2005_individualtop5Nmega2SL4Analysis

 

 

c)   For testing with bottom, average, and median test data

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_bottomall6.txt /input/megamillionprizesR.txt /output/2005to2005_bottomall6Analysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_individualbottomall6.txt /input/megamillionprizesR.txt /output/2005to2005_individualbottomall6Analysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_all6med.txt /input/megamillionprizesR.txt /output/2005to2005_all6medAnalysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_all6avg.txt /input/megamillionprizesR.txt /output/2005to2005_all6avgAnalysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_bottomall6R.txt /input/megamillionprizesR.txt /output/2005to2005_bottomall6Ranalysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_individualbottomall6R.txt /input/megamillionprizesR.txt /output/2005to2005_individualbottomall6RAnalysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_all6medR.txt /input/megamillionprizesR.txt /output/2005to2005_all6medRAnalysis

 

hadoop jar winningNumAnalysis.jar com.analysis.winningNumAnalysis /input/2015to2006.txt /input/2005to2005_all6avgR.txt /input/megamillionprizesR.txt /output/2005to2005_all6avgRAnalysis

 

4.5 Execution

There are 36 test analysis reports for each testing period, and there are 10 testing periods.  Therefore, there are 360 total test analysis reports together.  I can only show one as an example of the output to show the result.

Here is the detailed of one of those 360 reports as listed below:

File name: /output/2005to2005_top5NmegaAnalysis

Command: hdfs dfs -cat /output/2005to2005_top5NmegaAnalysis/part-r-00000

File content:

OpenJDK Server VM warning: You have loaded library /home/notroot/lab/software/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

15/09/08 16:27:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

ComparisonNumber  40

ComparisonNumber  7

ComparisonNumber  14

ComparisonNumber  5

ComparisonNumber  20

ComparisonNumber  36

WinningPrize 84000000

WinningPrize 373934

WinningPrize 5004

WinningPrize 718

WinningPrize 54

WinningPrize 9

WinningPrize 6

WinningPrize 2

WinningPrize 1

Winningnumbers        15        19        20        32        38        21

Winningnumbers        8          11        28        37        53        12

Winningnumbers        7          27        32        37        38        30

Winningnumbers        5          21        27        44        53        36

Winningnumbers        24        32        37        39        40        44

Winningnumbers        18        22        28        44        53        46

Winningnumbers        31        34        36        51        55        4

Winningnumbers        2          7          8          18        29        46

 

                               .

                               .

                               .

 

Winningnumbers   12   23   33   47   50   3

Winningnumbers   11   17   34   43   50   15

Winningnumbers   33   50   64   71   72   9

Won 1 number and 1 mega number!     5   21   27   44   53       

36

Won 1 number and 1 mega number!     14   23   27   36   45       

36

Won 1 mega number!     2   13   28   34   45   36

Won 1 mega number!     18   21   35   51   53   36

Won 1 mega number!     16   24   41   43   54   36

Won 1 number and 1 mega number!     8   9   14   38   44       

36

Won 1 mega number!     1   22   33   43   52   36

Won 1 number and 1 mega number!     6   13   14   31   36       

36

Won 1 mega number!     9   30   35   39   49   36

Won 1 mega number!     6   39   45   46   48   36

Won 1 mega number!     3   6   43   51   52   36

Won 1 mega number!     1   22   39   42   46   36

Won 1 mega number!     8   18   45   47   50   36

Won 1 mega number!     6   17   24   43   55   36

Won 1 number and 1 mega number!     4   13   20   29   48       

36

Won 1 mega number!     23   27   33   44   46   36

Won 3 numbers!     5   7   14   28   56   10

Won 1 mega number!     2   3   27   30   47   36

Won 1 number and 1 mega number!     21   27   32   40   52       

36

Won 1 number and 1 mega number!     27   31   39   40   46       

36

Won 1 mega number!     17   29   43   48   52   36

Won 1 mega number!     6   15   29   39   51   36

Won 1 number and 1 mega number!     5   9   22   36   49       

36

Won 1 number and 1 mega number!     5   24   26   29   53       

36

Won 1 mega number!     2   3   18   34   48   36

Won 3 numbers!     7   10   14   40   47   34

Won 1 mega number!     4   12   25   32   54   36

Won 3 numbers!     7   20   40   54   69   12

                                      Total Summary:

__________________________________________________________________

Any 6 numbers:                          0 is 0% of total # of tickets sold

Any 5 numbers:                                            0 is 0% of total # of tickets sold

 

Any 4 numbers and 1 mega number:            0 is 0% of total # of tickets sold

Any 4 numbers:                                0 is 0% of total # of tickets sold

Any 3 numbers and 1 mega number:            0 is 0% of total # of tickets sold

Any 3 numbers:                                3 is 0% of total # of tickets sold

Any 2 numbers and 1 mega number:            1 is 0% of total # of tickets sold

Any 2 numbers:                                45 is 5% of total # of tickets sold

Any 1 number and 1 mega number:            9 is 1% of total # of tickets sold

Any 1 number:                                 320 is 32% of total # of tickets sold

1 mega number:                                16 is 2% of total # of tickets sold

No number:                                    598 is 60% of total # of tickets sold

Profit or loss:                               $-925

Winning Percentage:                                       3%

 

 Click here to go to Chapter 5:   www.leisdesignservices.com/mapreducesummarydatarprt.htm

 Or click here to go to the Table of Content:  www.leisdesignservices.com/hadoopproofofconcept.htm