This file is indexed.

/usr/bin/tv_imdb is in xmltv-util 0.5.69-1.

This file is owned by root:root, with mode 0o755.

The actual contents of the file can be viewed below.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
#!/usr/bin/perl -w

=pod

=head1 NAME

tv_imdb - Augment XMLTV listings files with imdb.com data.

=head1 SYNOPSIS

tv_imdb --imdbdir <dir> [--help] [--quiet] [--download]
       [--prepStage (1-9,all)]

tv_imdb --imdbdir <dir> [--help] [--quiet]
       [--with-keywords] [--with-plot]
       [--movies-only] [--actors NUMBER]
       [--stats] [--debug]
       [--output FILE] [FILE...]

tv_imdb --imdbdir <dir>
        --validate-title 'movie title'
        --validate-year 2004
        [--with-keywords] [--with-plot]
        [--debug]

=head1 DESCRIPTION

Very similar to tv_cat in semantics (see tv_cat),
except whenever a programme appears with "date" entry the
title and date are used to look up extra data by using the
XMLTV::IMDB package.

B<--output FILE> write to FILE rather than standard output.

B<--with-keywords> include IDMb keywords in the output file.

B<--with-plot> include IDMb plot summary in the output file.

B<--actors NUMBER> number of actors from IMDb to add (default=3).

B<--quiet> disable all status messages (that normally appear on stderr).

B<--download> try to download data files if they are missing (in --prepStage).

B<--stats> output grab stats (stats output disabled in --quiet mode).

B<--debug> output info from movie matching

B<--movies-only> only augment programs that look like movie listings (4 digit
E<39>dateE<39> field).

All programs are checked against imdb.com data (unless --movies-only is used).

For the purposes of tv_imdb, an "exact" match is defined as a case
insensitive match against imdb.com data (which may or may not include the
transformation of E<39>&E<39> to E<39>andE<39> and vice-versa.

If the program includes a 4 digit E<39>dateE<39> field the following
matches are attempted, with the first successful match being used:

B<1.> an "exact" title/year match against movie titles is done

B<2.> an "exact" title match against tv series (and tv mini series)

B<3.> an "exact" title match against movie titles with production dates
within 2 years of the E<39>dateE<39> value.

Unless --movies-only is used, if the program does not include a 4 digit
E<39>dateE<39> field the following
matches are attempted, the first succeeding match is used:

B<1.> an "exact" title match against tv series (and tv mini series)

When a match is found in the imdb.com data the following is applied:

B<1.> the E<39>titleE<39> field is set to match exactly the title from the
imdb.com data. This includes modification of the case to match and any
transformations mentioned above.

B<2.> if the match is a movie, the E<39>dateE<39> field is set to imdb.com
4 digit year of production.

B<3.> the type of match found (Movie, TV Movie, Video Movie, TV Series,
or TV Mini Series) is placed in the E<39>categoriesE<39> field.

B<4.> the url to the www.imdb.com page is added

B<5.> the director is added if the match was a movie or if only one director
is listed in the imdb.com data (because some tv series have > 30 directors)

B<6.> the top 3 billing actors are added (use -actors [num] to adjust).

B<7.> genres added to E<39>categoriesE<39> field (current list of genres are
Short, Drama, Comedy, Documentary, Animation, Adult, Action, Family, Romance,
Crime, Thriller, Musical, Adventure, Western, Horror, Sci-Fi, Fantasy, Mystery,
War, Film-Noir, Music

B<8.> imdb user ratings added to E<39>star-ratingsE<39> field.

B<9.> imdb keywords added to E<39>keywordE<39> fields (if --with-keywords used).

B<10.> imdb plot summary is added (if --with-plot used).

=head1 HOWTO
In order to use tv_imdb, you need:

B<1.> choose a directory location to use for the tv_imdb database (youE<39>ll
need about 1 GB of free space),

B<2a.> run E<39>tv_imdb --imdbdir <dir> --prepStage all --downloadE<39>
to download the list files from imdb.com.  Or,

B<2b> If you have a slow network connection you may prefer to omit
the '--download' flag and be prompted for what you need to download by
hand.  See <http://www.imdb.com/interfaces> for the download sites.
Then once you have the files rerun without '--download'.

Note: '--prepStage' sucks a bit of memeory, but you can run each
prepStage separately by running --prepStage with each of the stages
(see --help for details).

B<3.> Once you have the database loaded try
E<39>cat tv.xml | tv_imdb --imdbdir <dir> > tv1.xmlE<39>.

Feel free to report any problems with these steps to xmltv-devel@lists.sf.net.

=head1 TESTING

The --validate-title and --validate-year flags can be used to validate the
information in the tv_imdb database. For exmple:

   tv_imdb --imdbdir . --validate-title 'Army of Darness' --validate-year 1994

=head1 BUGS

The '--prepStage' needs a lot of memory to run at a reasonable speed,
over 250 megabytes with the current imdb data files.  For there to be
250 megabytes free for tv_imdb, the system will need at least 512 megabytes
of RAM.  Running with less can take hours (or days!) - although fortunately
this stage needs to be run only once after downloading the data files.

Could use a --configure step just like the grabbers so you do not have
to specify the --imdbdir on the command line every time. Also this could
step you through the prep stages with more description of what is being
done and what is required. Configure could also control the number of
actors to add (since some movies have an awful lot), currently we are
adding the top 3.

How and what to look up needs to be option driven.

Needs some more controls for fine tuning "close" matches. For
instance, currently it looks like the North America grabber only has
date entries for movies, but the imdb.com data contains made for video
movies as well as as real movies, so itE<39>s possible to get the
wrong data to be inserted. In this case we may want to say "ignore tv
series" and "ignore tv mini series".  Along with this, weE<39>d want
to define what a "close" match is. For instance does a movie by the
same title with a date out by 1 year or 2 years considered a match
(currently weE<39>re using 2).

Nice to haves include: verification/addition of programe MPAA/VCHIP ratings,
addition of imdb.com user ratings (by votes) to programes. Potenially we
could expand to include "country of origin", "description", "writer" and
"producer" credits, maybe even "commentator".

Heh, if the XMLTV.dtd supported it, we could even include urls to head
shots of the actors :)

=head1 SEE ALSO

L<xmltv(5)>

=head1 AUTHOR

Jerry Veldhuis, jerry@matilda.com

=cut

use strict;
use XMLTV::Version '$Id: tv_imdb,v 1.35 2015/07/12 00:46:37 knowledgejunkie Exp $ ';
use Data::Dumper;
use Getopt::Long;

use XMLTV;
use XMLTV::Data::Recursive::Encode;
use XMLTV::Usage <<END
$0: augment listings with data from imdb.com
$0 --imdbdir <dir> [--help] [--quiet] [--download] [--prepStage (1-9,all)]
$0 --imdbdir <dir> [--help] [--quiet] [--download] [--with-keywords] [--with-plot] [--movies-only] [--actors NUMBER] [--stats] [--debug] [--output FILE] [FILE...]

END
;
use XMLTV::IMDB;

my ($opt_help,
    $opt_output,
    $opt_prepStage,
    $opt_imdbDir,
    $opt_quiet,
    $opt_download,
    $opt_stats,
    $opt_debug,
    $opt_movies_only,
    $opt_with_keywords,
    $opt_with_plot,
    $opt_num_actors,
    $opt_validate_title,
    $opt_validate_year,
   );

GetOptions('help' => \$opt_help,
	   'output=s' => \$opt_output,
	   'prepStage=s' => \$opt_prepStage,
	   'imdbdir=s' => \$opt_imdbDir,
	   'with-keywords' => \$opt_with_keywords,
	   'with-plot'     => \$opt_with_plot,
	   'movies-only' => \$opt_movies_only,
	   'actors=s'      => \$opt_num_actors,
	   'quiet' => \$opt_quiet,
	   'download' => \$opt_download,
	   'stats'         => \$opt_stats,
	   'debug+'         => \$opt_debug,
	   'validate-title=s' => \$opt_validate_title,
	   'validate-year=s'  => \$opt_validate_year,
	   ) or usage(0);

usage(1) if $opt_help;
usage(1) if ( not defined($opt_imdbDir) );

$opt_with_keywords=0 if ( !defined($opt_with_keywords) );
$opt_with_plot=0     if ( !defined($opt_with_plot) );
$opt_num_actors=3    if ( !defined($opt_num_actors) );
$opt_movies_only=0 if ( !defined($opt_movies_only) );
$opt_debug=0         if ( !defined($opt_debug) );

$opt_quiet=(defined($opt_quiet));
if ( !defined($opt_stats) ) {
    $opt_stats=!$opt_quiet;
}
else {
    $opt_stats=(defined($opt_stats));
}
$opt_debug=0  if $opt_quiet;

if ( defined($opt_prepStage) ) {
    print STDERR <<END
Building indices.  Be warned, this needs a lot of memory for the final stage
(working set about 250 megabytes).

END
    if ( ! $opt_quiet ) ;

    my %options =
      ('imdbDir' => $opt_imdbDir,
       'verbose' => !$opt_quiet,
       'showProgressBar' => !$opt_quiet,
       'stageToRun' => $opt_prepStage,
       'downloadMissingFiles' => $opt_download,
      );

    if ( $opt_prepStage eq "all" ) {
	for (my $stage=1 ; $stage <= 9 ; $stage++ ) {
	    my $n=new XMLTV::IMDB::Crunch(%options);
	    if ( !$n ) {
		exit(1);
	    }
	    my $ret=$n->crunchStage($stage);
	    if ( $ret != 0 ) {
		exit($ret);
	    }
	}
	print STDERR "database load complete, let the games begin !\n" if ( !$opt_quiet);
	exit(0);
    }
    else {
	my $n=new XMLTV::IMDB::Crunch(%options);
	if ( !$n ) {
	    exit(1);
	}
	my $ret=$n->crunchStage(int($opt_prepStage));
	if ( $ret == 0 && int($opt_prepStage) == 9 ) {
	    print STDERR "database load complete, let the games begin !\n" if ( !$opt_quiet);
	}
	exit($ret);
    }
}
elsif ( $opt_download ) {
    my %options =
    ('imdbDir' => $opt_imdbDir,
     'verbose' => !$opt_quiet,
     'showProgressBar' => !$opt_quiet,
     'stageToRun' => 'all',
     'downloadMissingFiles' => $opt_download,
    );

    my $n=new XMLTV::IMDB::Crunch(%options);
    if ( !$n ) {
	exit(1);
    }
    exit(0);
}

my $imdb=new XMLTV::IMDB('imdbDir' => $opt_imdbDir,
			 'verbose'         => $opt_debug,
			 'cacheLookups' => 1,
			 'cacheLookupSize' => 1000,
			 'updateKeywords'  => $opt_with_keywords,
			 'updatePlot'      => $opt_with_plot,
			 'numActors'       => $opt_num_actors,
			 );

#$imdb->{verbose}++;

if ( my $errline=$imdb->sanityCheckDatabase() ) {
    print STDERR "$errline";
    print STDERR "tv_imdb: you need to use --prepStage to rebuild\n";
    exit(1);
}

if ( !$imdb->openMovieIndex() ) {
    print STDERR "tv_imdb: open database failed\n";
    exit(1);
}

if ( defined($opt_validate_title) != defined($opt_validate_year) ) {
    print STDERR "tv_imdb: both --validate-title and --validate-year must be used together\n";
    exit(1);
}

if ( defined($opt_validate_title) && defined($opt_validate_year)  ) {
    my $prog;

    $prog->{title}->[0]->[0]=$opt_validate_title;
    $prog->{date}=$opt_validate_year;
    $imdb->{updateTitles}=0;

    #print Dumper($prog);
    my $n=$imdb->augmentProgram($prog, $opt_movies_only);
    if ( $n ) {
	$Data::Dumper::Sortkeys = 1; # ensure consistent order of dumped hash
	#my $encoding;
	#my $w = new XMLTV::Writer((), encoding => $encoding);
	#$w->start(shift);
	#$w->write_programme($n);
	print Dumper($n);
	#$w->end();
    }
    $imdb->closeMovieIndex();
    exit(0);
}

# test that movie database works okay
my %w_args = ();
if (defined $opt_output) {
    my $fh = new IO::File ">$opt_output";
    die "cannot write to $opt_output\n" if not $fh;
    %w_args = (OUTPUT => $fh);
}

my $numberOfSeenChannels=0;

my $w;
my $encoding;   # store encoding of input file

sub encoding_cb( $ ) {
    die if defined $w;
    $encoding = shift;    # callback returns the file's encoding
    $w = new XMLTV::Writer(%w_args, encoding => $encoding);
}

sub credits_cb( $ ) {
    $w->start(shift);
}

my %seen_ch;
sub channel_cb( $ ) {
    my $c = shift;
    my $id = $c->{id};
    $Data::Dumper::Sortkeys = 1; # ensure consistent order of dumped hash
    if (not defined $seen_ch{$id}) {
	$w->write_channel($c);
	$seen_ch{$id} = $c;
	$numberOfSeenChannels++;
    }
    elsif (Dumper($seen_ch{$id}) eq Dumper($c)) {
	# They're identical, okay.
    }
    else {
	warn "channel $id may differ between two files, "
	  . "picking one arbitrarily\n";
    }
}

sub programme_cb( $ ) {
    my $prog=shift;

    # The database made by IMDB.pm is read as iso-8859-1. The xml file may be different (e.g. utf-8).
    # IMDB::augmentProgram does not re-encode the data it adds, so the output file has invalid characters (bug #440).

    my $orig_prog;
    if (lc($encoding) ne 'iso-8859-1') {
      # decode the incoming programme
      $orig_prog = $prog;
      $prog = XMLTV::Data::Recursive::Encode->decode($encoding, $prog);
    }

    # augmentProgram will now add imdb data as iso-8859-1
    my $nprog=$imdb->augmentProgram($prog, $opt_movies_only);
    if ( $nprog ) {
      if (lc($encoding) ne 'iso-8859-1') {
        # re-code the modified programme back to original encoding
        $nprog = XMLTV::Data::Recursive::Encode->encode($encoding, $nprog);
      }
	$prog=$nprog;
    }
    else {
        $prog = $orig_prog;
    }

    # we only add movie information to programmes
    # that have a 'date' element defined (since we need
    # a year to work with when verifing we got the correct
    # hit in the imdb data)
    $w->write_programme($prog);
}

@ARGV = ('-') if not @ARGV;

XMLTV::parsefiles_callback(\&encoding_cb, \&credits_cb,
			   \&channel_cb, \&programme_cb,
			   @ARGV);
# we only get a Writer if the encoding callback gets called
if ( $w ) {
    $w->end();
}

if ( $opt_stats ) {
    print STDERR $imdb->getStatsLines($numberOfSeenChannels);
}
$imdb->closeMovieIndex();
exit(0);