l i n u x - u s e r s - g r o u p - o f - d a v i s
Next Meeting:
March 3: Social gathering
Next Installfest:
Latest News:
Feb. 5: LUGOD seeking speakers for 2015
Page last updated:
2002 Aug 09 16:01

The following is an archive of a post made to our 'vox-tech mailing list' by one of its subscribers.

Report this post as spam:

(Enter your email address)
Re: [vox-tech] shell script challenge
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [vox-tech] shell script challenge

(pretending this hasn't been beaten to death...)

not sure why, but i looked at the man page for sort and uniq.
there are options for each that solve your problem.  here's an example:

  find /some/path -type f >! /tmp/f
  cat /tmp/f | xargs md5sum >! /tmp/g
  cat /tmp/g | sort -k1,32 | uniq -w 32 -D

- i'm writing temp files because i presume you're doing this to
  lots of files, and the shell will quickly get overloaded with cmdline
  wildcard arguments.  furthermore, it gives you an intermediate product
  to inspect.
- xargs is a cool filter to handle situations where you would have
  lots of cmdline arguments (like filenames)
- the -k option to sort tells it to use characters 1-32 (in this case,
  the md5 hash value) as the key
- the -w option to uniq tells it to compare no more than the first
  32 characters; the -D option displays all the duplicates for any
  repeated line
- i recognize there are more concise/elegant ways to do this...

as i was trying this out on some detritus i found some surprising matches.
in a few cases, turned out to be short configuration-type files.  got me
to thinking, i don't think one would *want* to delete ALL duplicate files
on a system.  i understand the intent, but it could be dangerous to
do so (might not notice it right away).  for example, different packages
could use an identical file, e.g., a configuration file, in a different
location; wouldn't want to delete those!  anyway, seems to me that
manual inspection is required, pre-deletion, as a sanity check.


On Wednesday 07 August 2002 12:20 pm, Chris McKenzie wrote:
> I'm using cygwin and I was given the request by my boss to remove all
> duplicate files from the server
> the server is on the x: drive of the windows machine which means that
> cygwin saw it as /cygwin/x
> I forget exactly what command I ran toget checksums.txt
> but it is in the format
> <checksum> *x:<filename>
> The challenge is to find the duplicate checksums and print the file name
> of those checksums.  This is tricky because the directories contain spaces
> which gawk, sed, etc ... see as fields.  Even if I change the IFS to * and
> then use gawk to print the *x:<fname> <checksum> -- sort wouldn't know how
> to deal with it which would make uniq useless (I think).  if I do it the
> other way, <checksum> *x:<filename> sort will work fine but uniq will fail
> because the filename is there.  if I exclude the filename with a gawk ' {
> print $1 } ' then sort and uniq will work fine but I won't have a
> filename.  So all the combinations I can think of fail.  Does anyone know
> how I can find only the duplicate checksums and the file names associated.
> **I realize that with a lbut the problem is that there are 4,575 duplicate
> checksums using:
> cat checksums.txt | awk ' { print $1 } ' | sort -uniq -d | wc -l
> and 46340 files on the server, which seems like it would take an awful
> long time.  any suggestions?
> Sincerely,
> 	Christopher J. McKenzie
> 	cjm@ucdavis.edu
> 	mckenzie@cs.ucdavis.edu
> 	H: (818) 991-7724
> 	C: (818) 429-3772
> 	1815 Mesa Ridge Ave
> 	Westlake Village, CA 91362
> _______________________________________________
> vox-tech mailing list
> vox-tech@lists.lugod.org
> http://lists.lugod.org/mailman/listinfo/vox-tech

vox-tech mailing list

LUGOD Group on LinkedIn
Sign up for LUGOD event announcements
Your email address:
LUGOD Group on Facebook
'Like' LUGOD on Facebook:

Hosting provided by:
Sunset Systems
Sunset Systems offers preconfigured Linux systems, remote system administration and custom software development.

LUGOD: Linux Users' Group of Davis
PO Box 2082, Davis, CA 95617
Contact Us

LUGOD is a 501(c)7 non-profit organization
based in Davis, California
and serving the Sacramento area.
"Linux" is a trademark of Linus Torvalds.

Sponsored in part by:
Appahost Applications
For a significant contribution towards our projector, and a generous donation to allow us to continue meeting at the Davis Library.