The History Of Data Deduplication

October 11, 2018

HTC - Data Recovery @ Wierd

Data deduplication has been around, at least in its most primitive form, since the 1970’s. It initially started because companies wanted to store large amount of customer contact information without using a large amount of storage space. One of the first ideas was to go through and remove duplicate data. For example, a company might have an address for shipping, and an address for billing to a given customer. In these cases, those identical addresses would be combined into one file. This was done by data entry clerks who would review the data line by line and get rid of duplicates.

Of course, the amount of personnel needed to do this was extensive and it took a very long time. Sometimes, the data deduplication process would take months to complete. However, considering that most of this occurred on hard copy, it wasn’t a major problem. The big problems cam along when computer use became widespread in office environments.

With computers in wide use and the explosion of the internet, the amount of data available exploded as well. Backup systems were created to ensure that companies would not lose all their data. As time went by, floppy discs and other external hardware was used to store this data. Unfortunately, this data would soon fill up these discs and the amount of space to store this data was extensive.

With cloud storage and other alternative storage options, companies began moving their storage to a virtual environment. They also moved to disk-based storage over tape based, simply because it was low-cost and required less space. However, these storage options were expensive and difficult to manage as data grew. The same data would get saved over and over again. This redundant data was not needed and took up valuable storage space.

Companies might have customized their backup plans to eliminate duplication, but there was no fast way to do this. That is when IT professionals began working on algorithms to automate the data deduplication process. They generally did this on a cases by case basis, with their goal to optimize their own backup files. Their algorithms would be customized to meet their own individual needs.

There was no one company that came up with the idea of data deduplication. Instead, the need to find a way to reduce duplicate files was a common need in the industry. There were many computer scientists who advanced data deduplication technology significantly, but there is no one scientist who was solely responsible for it. While many have claimed credit for the term ‘data deduplication’ itself, no one person can claim credit for the idea itself.

Instead, the creation of data deduplication algorithms was more of a compilation. People in the IT industry saw a need to reduce data duplicates and they filled that need to reduce those duplicated file by creating algorithms. As data increases, people will continue to find ways to compress data in a way that makes it easy to store.

20 Comments

  • ramesh babu October 11, 2018 at 7:19 pm

    శ్రీధర్ గారు నమస్కారం
    సార్ నా దగ్గర వెస్ట్రన్ డిజిటల్ ఎక్సటర్నల్ హార్డ్ డిస్క్ ఉంది. దాన్ని నా లాప్టాప్ కి కనెక్ట్ చేసినప్పుడు సి ఫోల్డర్ లో ఐకాన్ చూపించటం లేదు. కానీ డివైస్ అండ్ ప్రింటర్ ఆప్షన్ లో ఐకాన్ చూపిస్తుంది కానీ ఓపెన్ అవడం లేదు పరిస్కారం చెప్పండి.సమస్య ఏమిటో కూడా చెప్పండి ప్లీజ్.

  • Gopi Nadh October 11, 2018 at 7:19 pm

    I used Ease Us software. It also works in the same manner.

  • KiRama Chaitanya Guntur October 11, 2018 at 7:19 pm

    how to remove $recycle.bin and system volume information in windows 10 please help me

  • Nageswarao Studio October 11, 2018 at 7:19 pm

    yas easeus data recovery

  • bhanu October 11, 2018 at 7:19 pm

    sir drives lo files delete aithe 7data recovery tho files recovevery chestunte 7 data recovery suite registration code username adugudundi adiemito chepandi sir

  • JAGADEESH GAMING WORLD October 11, 2018 at 7:19 pm

    marrage delete ayi 3 months ayindi pls dhaniki samadham chepandi

  • JAGADEESH GAMING WORLD October 11, 2018 at 7:19 pm

    sir nenu video grapher naa sistem lo oka marrage delete ayindi dhani thiri phondhali dhaniki margamuundha

  • Krishna Upputella October 11, 2018 at 7:19 pm

    vadaleedu it is very good

  • SNR Msc,MEd October 11, 2018 at 7:19 pm

    sir which OS you used please tell me

  • nagesh photography October 11, 2018 at 7:19 pm

    psd vastaya bro

  • Manju Nath October 11, 2018 at 7:19 pm

    సార్ మెమరీ కార్డు ని రికవర్ చేస్తుంటే మై కంప్యూటర్ లో రిమూవల్ డిస్క్ చూపిస్తోంది దానిని ఓపెన్ చేస్తుంటే ప్లీజ్ ఇన్సర్ట్ డిస్క్ అని చూపిస్తోంది డానికి ఓపెన్ చేయడం ఎలా

  • Manju Nath October 11, 2018 at 7:19 pm

    సార్ మెమరీ కార్డు ని రికవర్ చేస్తుంటే మై కంప్యూటర్ లో రిమూవల్ డిస్క్ చూపిస్తోంది దానిని ఓపెన్ చేస్తుంటే ప్లీజ్ ఇన్సర్ట్ డిస్క్ అని చూపిస్తోంది డానికి ఓపెన్ చేయడం ఎలా

  • charan ch69 October 11, 2018 at 7:19 pm

    sir na COMPUTER lo D drive lo files anni kanipinchadam ledu. nenu konni rojulu chusukoledu. tarvata chuste anta khaliga kanpinchindi. idi jarigi 08 months outundi.
    mari na files nenu recovery chesukogalana dayachesi telupa galaru. avi maa babu vidies photos pleaz help me sir.

  • Bujji Prasad October 11, 2018 at 7:19 pm

    7 recovery use chesthunappudu license aduguthundi can u say how to download pendrive recovery software……

  • Venky Chowdary October 11, 2018 at 7:19 pm

    sir E app mana mobile lo install cheskunte work chestunda or pc only plz reply sir

  • emmanuel bartholomew October 11, 2018 at 7:19 pm

    nice

  • Pranay Tony October 11, 2018 at 7:19 pm

    Sir What about Piriform Recuva, I care Data Recovery Softwares? Itried it but didn't worked for me! What should I do?

  • varalakshmi Veerla October 11, 2018 at 7:19 pm

    sir nenu recovery deleted photos,files app dwnlod chesanu.kani na deleted photos recovery avvatledu

  • madabhushanam raghava chary October 11, 2018 at 7:19 pm

    not yet .i wiil try to recover of my external drive

  • Pravallika Pothula October 11, 2018 at 7:19 pm

    It's very useful thanku

Leave a Reply

Your email address will not be published. Required fields are marked *