Difference between data cleaning and data organization?

[maneesha]

The Introduction episode notes that one objective is:

• Differentiate data cleaning from data organization.

However, this is not covered in the introductino episode.

[brownsarahm]

Thanks @maneesha this is a good point that should be addressed.

[BhaswatiRoy]

Hello
I would like to work on this issue
It would be great if I could be assigned
Thanks

[bencomp]

Thanks for your offer, @BhaswatiRoy! Your contribution to this lesson would be much appreciated. I don't think anyone else is currently working on this, so feel free to start. I can still assign the issue to you.

Perhaps you can share your ideas for change(s) here first, so that we can help you go in the right direction with a PR? If you need any help, please let us know.

[BhaswatiRoy]

Hello @bencomp
After looking through the given link, I believe the "Motivations for the OpenRefine Lesson" section gives insights into the issue.
So the detailed differences between data cleaning & organization can be added under this section.
Or maybe a separate section can be created for explaining in depth.

[bencomp]

I think that section needs an update anyway (I mentioned it in a comment in issue #103 as well), so maybe it would be good to add a new short section explaining the differences. Do you have an idea of what you would add? I feel it does not need to be very detailed, because understanding this difference is probably not necessary to follow the rest of the lesson.

[BhaswatiRoy]

We can add the meaning of both terms along with a short example to explain how both of them actually work.

[bencomp]

That sounds good. I think core to this issue is the question what the actual differences are, so any suggestions for what to write are especially welcome.

[bencomp]

Let me make some suggestions for what data organisation and data cleaning entail. This is up for discussion.

Data organisation:

• combining data from multiple files
• reordering columns (supported! see Introduce menu for column reordering #83)
• moving table contents from a random position in a spreadsheet (not starting in cell A1) to the top left of the spreadsheet
• splitting or combining values into or from multiple columns

Data cleaning:

• removing unwanted variations, like typos, word order differences, date formats (e.g. ., / or - as separators)
• normalising capitalisation
• dealing with outliers
• normalising missing values

Following these example definitions, that I really just made up, it is not like OpenRefine supports only data cleaning and not data organisation. Maybe we should remove the objective? We can (or should) still add examples like these to better introduce what OpenRefine can and cannot do (as discussed in #86 and #103).

[bencomp]

Reminder to remove the mention of this issue from the Instructor note in the Introduction section when this issue is being resolved. See #183.