Conclusion

• There are a lot of jobs dealing with large data sets.
• Take Home Points
  1. There is a rapidly increasing amount of data to be processed.
  2. Standard databases can help with relatively large data (< 2 GB).
  3. You can not process really big data well with just one machine.
  4. For really big datamining tasks, you need distributed storage and distributed processing.
• Reading: For this week is practical advice for analysis of large complex data sets . You might also want to look at MAD Skills: New Analysis Practices for Big Data by Cohen, Dolan, Dunlap, Hellerstein and Welton.
• Reading: for next week is Russell and Norvig Making Simple Decisions Chapter (pp. 621-647) (particularly section 3).