This is error because the dataset which you are considering consist of na value. i.e null value
while performing matrix multipication and other any operation with null value create error. To overcome this problem you need to select one of the follow method: i) Either use rm=na while taking input. i.e mean remove null values. ii) Take the dataset which doesn't consist of null value. i.e clean dataset. No need to go through the document below: It is the reference i followed and search and paste for the future reference at that time. I saw high view in this page so i share my experience with you ppl. I am trying the below R script to built logistic regression model using RHadoop (rmr2, rhdfs packages) on an HDFS data file located at "hdfs://:/somnath/merged_train/part-m-00000" and then testing the model using a test HDFS data file at "hdfs://:/somnath/merged_test/part-m-00000". We are using CDH4 distribution with Yarn/MR2 running parallel to MR1 supported by Hadoop-0.20. And using the hadoop-0.20 mapreduce and hdfs versions to run the below RHadoop script as Sys.setenv commands shown below. However, whenever I am running the script, I am facing the below error with very little luck to bypass it. I would appreciate if somebody point me to the possible cause of this error which seems to be due to wrong way of lapply call in R without handling NA arguments. SAMPLE TRAIN DATASETSAMPLE TEST DATASET |
|||
Thursday, 30 October 2014
Error in FUN(X[[2L]], …) : Sorry, parameter type `NA' is ambiguous or not supported
Linear Regression in R Mapreduce(RHadoop)
I m new to RHadoop and also to RMR...
I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.
Tring to read the file from hdfs
Error:Code : |
|||||||||||||
|
Thursday, 9 October 2014
select
> example(select) select> #data.frame select> where(mtcars, cyl>4 & mpg > 15) mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6 select> #pipe select> as.data.frame(where(input(mtcars), cyl > 4 & mpg > 15)) mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6 select> # select two columns select> as.data.frame(transmute(input(mtcars), cyl, mpg)) cyl mpg 1 6 21.0 2 6 21.0 3 4 22.8 4 6 21.4 5 8 18.7 6 6 18.1 7 8 14.3 8 4 24.4 9 4 22.8 10 6 19.2 11 6 17.8 12 8 16.4 13 8 17.3 14 8 15.2 15 8 10.4 16 8 10.4 17 8 14.7 18 4 32.4 19 4 30.4 20 4 33.9 21 4 21.5 22 8 15.5 23 8 15.2 24 8 13.3 25 8 19.2 26 4 27.3 27 4 26.0 28 4 30.4 29 8 15.8 30 6 19.7 31 8 15.0 32 4 21.4 select> # create additional column select> as.data.frame(transmute(input(mtcars), ratio = cyl/mpg, .cbind = TRUE)) mpg cyl disp hp drat wt qsec vs am gear Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 carb ratio Mazda RX4 4 0.2857143 Mazda RX4 Wag 4 0.2857143 Datsun 710 1 0.1754386 Hornet 4 Drive 1 0.2803738 Hornet Sportabout 2 0.4278075 Valiant 1 0.3314917 Duster 360 4 0.5594406 Merc 240D 2 0.1639344 Merc 230 2 0.1754386 Merc 280 4 0.3125000 Merc 280C 4 0.3370787 Merc 450SE 3 0.4878049 Merc 450SL 3 0.4624277 Merc 450SLC 3 0.5263158 Cadillac Fleetwood 4 0.7692308 Lincoln Continental 4 0.7692308 Chrysler Imperial 4 0.5442177 Fiat 128 1 0.1234568 Honda Civic 2 0.1315789 Toyota Corolla 1 0.1179941 Toyota Corona 1 0.1860465 Dodge Challenger 2 0.5161290 AMC Javelin 2 0.5263158 Camaro Z28 4 0.6015038 Pontiac Firebird 2 0.4166667 Fiat X1-9 1 0.1465201 Porsche 914-2 2 0.1538462 Lotus Europa 2 0.1315789 Ford Pantera L 4 0.5063291 Ferrari Dino 6 0.3045685 Maserati Bora 8 0.5333333 Volvo 142E 2 0.1869159 select> # summaries select> as.data.frame(transmute(input(mtcars), mean(cyl), mean(mpg))) mean.cyl. mean.mpg. 1 6.1875 20.09062 select> # summaries by groups select> as.data.frame(transmute(group(input(mtcars), cyl), mean(mpg))) cyl mean.mpg. 1 6 19.74286 1.1 4 26.66364 1.2 8 15.10000
Friday, 12 September 2014
what tools can one use on data that has so many missing values?
Hey all, what tools can one use on data that has so many missing values?
Comments
11 comments
Jump to most recent comment
-
DavidAssociate Planner at Kittelson & Associates, Inc.What do you mean by "so many"? If you're talking about more than half the data set, then you may be out of luck. If you're talking in the range up to 20% missing, there are a number of good references on dealing with missing data. The classic is Little & Rubin, Statistical Analysis with Missing Data (2nd ed.). There's a new book out by van Buuren, Flexible Imputation of Missing Data. I suggest reading up first on the different types of missing data mechanisms; if the data are missing completely at random, you're pretty safe. Dealing with other missing data mechanisms will take a bit more caution. The book by McLachlan and Krishnan on the EM algorithm may also be worth a look, but you can find information on the EM algorithm in a lot of books and papers these days. Good luck!Petar O., Dainius T. and 3 others like this -
MarkConsultant - Rescuer of Doomed Projects; Solver of Impossible Problems; Inspired by Sharing How to Do It AllDorcas,David R., Tymoteusz W. and 2 others like this
Gelman et al, "Bayesian Data Analysis," also has a great chapter on missing data, and will be consistent with the references David recommended.
Mark Powell -
DorcasMathematical Modelling Intern at ICIPE - International Centre of Insect Physiology and EcologyThank you all for the response. David and Mark;where can I get the book? -
MarkConsultant - Rescuer of Doomed Projects; Solver of Impossible Problems; Inspired by Sharing How to Do It AllDorcas,
I usually just go to Amazon.com. All of the references suggested by me and David are available there.
Mark Powell -
MarxLecturer at Midlands State UniversityFor missing data treatment especially for credit scoring purpose I would suggest the Bayesian Inference with missing data using Bound and Collapse Method for nonignorable missing data mechanism. Bound and Collapse method is a deterministic imputation model based on Dirichlet probability distribution of a multinomial random variable was suggested by Ramoni and Sebastian, 1998) and applied to credit scoring of SMEs by Chen and Astebro (2003) and proved very handy. The maximum likelihood can also does the tricks as long as the missingness can be modeled.
-
TymoteuszResearch and Data Analytics SpecialistIn my opinion Gelman's et al. book is worth consideration. In another book by Gelman and Hill on regression (http://www.stat.columbia.edu/~gelman/arm/) there's also a chapter on missing data, you can find this one online:
http://www.stat.columbia.edu/~gelman/arm/missing.pdf
You could check also "The BUGS Book" by Lunn et al ( http://www.amazon.co.uk/BUGS-Book-Practical-Introduction-Statistical/dp/1584888490/ ) it also gives some hints on missing data in Bayesian approach. -
DavidAssociate Planner at Kittelson & Associates, Inc.Gelman et al. latest edition is worth buying in its own right: the best book on Bayesian data analysis is now even better. If your book budget is limited and if you want to know how to handle missing data I recommend Gelman et al. first (covers more than just missing data), then van Buuren (most up-to-date & more complete coverage of missing data than Gelman et al.), with Little & Rubin coming in third.
-
MarkConsultant - Rescuer of Doomed Projects; Solver of Impossible Problems; Inspired by Sharing How to Do It AllDavid,
You made a comment above " If you're talking about more than half the data set, then you may be out of luck."
It really depends on what the data are and how the missing data are "missing." You can get good usable estimates with 100% of the data missing if you have the right kind of data and the right kind of "missing." It is all in the formulation of the likelihood.
Mark Powell -
MarkConsultant - Rescuer of Doomed Projects; Solver of Impossible Problems; Inspired by Sharing How to Do It AllDavid,
Let me correct myself, you can always do a prior predictive estimate without any data at all, even without any missing data.
Mark Powell -
ThomasScientist at JOANNEUM RESEARCH Forschungsgesellschaft mbHI like the R package VIM: Visualization and Imputation of Missing Values (http://cran.r-project.org/web/packages/VIM/index.html). Especially the graphics can really help on understanding the structure of missing values. The package also comes with an optional graphical user interface.
There are also some good manuals and explanations around in the web (e.g. http://www.unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.44/2014/mtg1/Topic_5_Austria.pdf). -
TymoteuszResearch and Data Analytics SpecialistBtw, do any you have any sources to recommend on missing (at random) data in directed social network? Thanks.
Thursday, 28 August 2014
A Method of Grouping and Summarizing Data of Big Text Files in R Language
It is common to use R language to group and summarize data of files.
Sometimes we may find ourselves processing comparatively big files which
have smaller computed result and bigger source data. We cannot load
them wholly to the memory when we need to compute them. The only
solutions could be batch importing and computing as well as result
merging. We’ll use an example in the following to illustrate the way of R
language to group and summarize data from big text files.
Here is a file, sales.txt, of 1G size, which contains a great number of records of sales orders. We want to group field CLIENT and summarize field AMOUNT. “\t” is used in the file as the column separator. The first rows of data are as follows:
R’s solution:
Part of the computed result:
Code interpretation:
The 1st line: Open the file handle.
The 2nd ~ 3rd line: Import the first batch of 100,000 rows of data, group and summarize them and save the result in result.
The 4th ~ 8th line: Import data by loop, with 100,000 rows of data per batch, and store them in the variable databatch. Then get the second and fourth field, i.e. “CLIENT” and “AMOUNT”, merge databatch into result, and execute grouping operation.
It can be seen that, at a certain moment, only databatch, which includes 100,000 rows of data, and result, the summarizing result, have memory usage. Usually, the size of the latter is small and will not result in a memory overflow.
The 11th line: Close the file handle.
Matters needing attention:
Data frame. Because the data frame of R language cannot directly perform the computing of big files, loop statement is necessary to help to do the job in this occasion. The steps are: import a batch of data and merge them into the data frame result; group and summarize result and then import the next batch of data. You can see that this part of code of loop statement is a little complicated.
Column name. As the first row of data is the column name, header=TRUE can be used in the first batch of data to directly set the column name. But the subsequent data hasn’t column names and header=FALSE should be used to import data. The default column names are V1, V2 and so forth when header=FALSE is used. But the default column names are Group.1 and x after grouping and summarizing are executed, and col.names is needed to change the column names in order to maintain structure consistency both before and after grouping and summarizing and set the stage for the subsequent merging. The code about column names is worth our notice because it is easy to get wrong.
esProc can process data in batches automatically, which requires no manual control from the programmers by loop statement and produces quite simple code:
Cursor is a data type used for structured data computing in esProc. Its usage is similar to that of the data frame, but it is better at processing big files and performing complicated computations. What’s more, @t option in the code indicates that the first line of the file is the column name. So it is convenient to use the column name directly in subsequent computation.
Python’s code structure, which also requires manual loop control, is similar to that of R language. But Python itself hasn’t the structured data type, like data frame or cursor, so its code is executed in a lower level:
Except for the above two-dimensional array, Python can execute the operation with the third-party packages. For example, pandas has the structured data object similar to the data frame. pandas simplifies the code in a similar way as R language. But it lacks sufficient ability to perform big file computing, thus loop statement is still needed while programming.
Here is a file, sales.txt, of 1G size, which contains a great number of records of sales orders. We want to group field CLIENT and summarize field AMOUNT. “\t” is used in the file as the column separator. The first rows of data are as follows:
| ORDERID CLIENT SELLERID AMOUNT ORDERDATE 1 WVF Vip 5 440.0 2009-02-03 2 UFS Com 13 1863.4 2009-07-05 3 SWFR 2 1813.0 2009-07-08 4 JFS Pep 27 670.8 2009-07-08 5 DSG 15 3730.0 2009-07-09 6 JFE 10 1444.8 2009-07-10 7 OLF 16 625.2 2009-07-11 |
R’s solution:
| 1 | con <- file("E: \\sales.txt", "r") |
| 2 | result=read.table(con,nrows=100000,sep="\t",header=TRUE) |
| 3 | result<-aggregate(result[,4],list(result[,2]),sum) |
| 4 | while(nrow(databatch<-read.table(con,header=FALSE,nrows=100000,sep="\t",col.names=c("ORDERID","Group.1","SELLERID","x","ORDERDATE")))!=0)
{ |
| 5 | databatch<-databatch[,c(2,4)] |
| 6 | result<-rbind(result,databatch) |
| 7 | result<-aggregate(result[,2],list(result[,1]),sum) |
| 8 | } |
9
|
close(con) |
Group.1 x 1 ARO 17981798 2 BDR 85584558 3 BON 51293129 4 BSF 287908788 5 CHO 23482348 |
Code interpretation:
The 1st line: Open the file handle.
The 2nd ~ 3rd line: Import the first batch of 100,000 rows of data, group and summarize them and save the result in result.
The 4th ~ 8th line: Import data by loop, with 100,000 rows of data per batch, and store them in the variable databatch. Then get the second and fourth field, i.e. “CLIENT” and “AMOUNT”, merge databatch into result, and execute grouping operation.
It can be seen that, at a certain moment, only databatch, which includes 100,000 rows of data, and result, the summarizing result, have memory usage. Usually, the size of the latter is small and will not result in a memory overflow.
The 11th line: Close the file handle.
Matters needing attention:
Data frame. Because the data frame of R language cannot directly perform the computing of big files, loop statement is necessary to help to do the job in this occasion. The steps are: import a batch of data and merge them into the data frame result; group and summarize result and then import the next batch of data. You can see that this part of code of loop statement is a little complicated.
Column name. As the first row of data is the column name, header=TRUE can be used in the first batch of data to directly set the column name. But the subsequent data hasn’t column names and header=FALSE should be used to import data. The default column names are V1, V2 and so forth when header=FALSE is used. But the default column names are Group.1 and x after grouping and summarizing are executed, and col.names is needed to change the column names in order to maintain structure consistency both before and after grouping and summarizing and set the stage for the subsequent merging. The code about column names is worth our notice because it is easy to get wrong.
Alternative solutions:
Python, esProc and Perl can also perform the same operation. They can
execute the grouping and summarizing of data from big text files and
the subsequent structured data computing as R language does. We’ll
briefly introduce the coding methods used by esProc and Python.esProc can process data in batches automatically, which requires no manual control from the programmers by loop statement and produces quite simple code:
A
|
|
1
|
=file("e:/sales.txt").cursor@t()
|
2
|
=A2.groups(CLIENT;sum(AMOUNT))
|
Cursor is a data type used for structured data computing in esProc. Its usage is similar to that of the data frame, but it is better at processing big files and performing complicated computations. What’s more, @t option in the code indicates that the first line of the file is the column name. So it is convenient to use the column name directly in subsequent computation.
Python’s code structure, which also requires manual loop control, is similar to that of R language. But Python itself hasn’t the structured data type, like data frame or cursor, so its code is executed in a lower level:
| 1 | from itertools import groupby |
| 2 | from operator import itemgetter |
| 3 | result = [] |
| 4 | myfile = open("E:\\sales.txt",'r') |
| 5 | BUFSIZE = 10240000 |
| 6 | myfile.readline() |
| 7 | lines = myfile.readlines(BUFSIZE) |
| 8 | value=0 |
| 9 | while lines: |
| 10 | for line in lines: |
| 11 | record=line.split('\t') |
| 12 | result.append([record[1],float(record[3])]) |
| 13 | result=sorted(result,key=lambda x:(x[0])) # the sorting before grouping is executed |
14
|
batch=[] |
15
|
for key, items in groupby(result, itemgetter(0)): # group using groupBy function |
16
|
value=0 |
17
|
for subItem in items:value+=subItem[1] |
18
|
batch.append([key,value]) # finally, merger the summarizing results into a two-dimensional array |
19
|
result=batch |
20
|
lines = myfile.readlines(BUFSIZE) |
21
|
myfile.close() |
Except for the above two-dimensional array, Python can execute the operation with the third-party packages. For example, pandas has the structured data object similar to the data frame. pandas simplifies the code in a similar way as R language. But it lacks sufficient ability to perform big file computing, thus loop statement is still needed while programming.
Tuesday, 11 March 2014
How to upgrade R in Windows
1. Install package installr. For this in Rstudio u can click tools and install packages ... and type installr.
Or From command line:- >install.packages("installr")
2. Now type >require(installr)
3. updateR()
#This will only work AFTER R 3.0.0 will be release to your CRAN mirror
Or From command line:- >install.packages("installr")
2. Now type >require(installr)
3. updateR()
#This will only work AFTER R 3.0.0 will be release to your CRAN mirror
Subscribe to:
Posts (Atom)
mapreduce(input = as.matrix(hdfs.read.text.file(r.file)),map = function(.,M) { keyval(1,lapply(as.numeric(M[,-1] %*% t(weight)), function(z) 1/(1 + exp(-z))))} )))where the input to map is a matrix M read from a file stored in HDFS. Most probably the call to lapply may not be getting the expected input from the matrix. I have added sample train and test data inputs from HDFS files to explain better – somnathchakrabarti Aug 11 at 8:57traceback(),debug(),debugonce()andbrowser()can yield insightful. – Roman Luštrik Aug 11 at 10:25