dkim2.t failures

[Justin Mason]

2009/7/2 Mark Martinec <Mark.Martinec+sa@...>:

> On Thu, 02 Jul 2009 00:13:52 +0200, Kevin A. McGrail <kmcgrail@...>
> wrote:
>
>> I'm running long tests and net tests and getting a lot of failures in the
>> DKIM area.  I'm running 0.28.
>>
>> Is this a known issue?  Should I upgrade DKIM and try again which would
>> like make me suggest a higher version requirement for the makefile.
>
> Don't know, it passes all tests for me.
> Does it happen to others too?

yep, me too.

> Is the plugin enabled at all?
> I would expect at least the tests for a presence of a signature
> (the DKIM_SIGNED) should work regardless of the version of Mail::DKIM.
>
> The DKIM_VALID would probably fail in some tests with older
> version on Mail::DKIM, but most of them should still pass.

here's what I see (with Mail::DKIM 0.19):

t/dkim2....1..80
# Running under perl version 5.008008 for linux
# Current time local: Thu Jul  2 11:05:07 2009
# Current time GMT:   Thu Jul  2 11:05:07 2009
# Using Test.pm version 1.25
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/l
ocalrules.tmp -p log/test_default.cf  -t < data/dkim/test-pass-16.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 1 in t/SATest.pm at line 707
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 2 in t/SATest.pm at line 707 fail #2
Output can be examined in: log/d.dkim/1
# Failed test 3 in t/dkim2.t at line 50
#  t/dkim2.t line 50 is:   ok ok_all_patterns();
        Checking DKIM_SIGNED
not ok 1
        Checking DKIM_VALID
not ok 2
not ok 3
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/l
ocalrules.tmp -p log/test_default.cf  -t < data/dkim/test-pass-11.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 4 in t/SATest.pm at line 707 fail #3
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 5 in t/SATest.pm at line 707 fail #4
Output can be examined in: log/d.dkim/4
# Failed test 6 in t/dkim2.t at line 50 fail #2
        Checking DKIM_SIGNED
not ok 4
        Checking DKIM_VALID
not ok 5
not ok 6
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/l
ocalrules.tmp -p log/test_default.cf  -t < data/dkim/test-pass-08.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 7 in t/SATest.pm at line 707 fail #5
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 8 in t/SATest.pm at line 707 fail #6
Output can be examined in: log/d.dkim/7
# Failed test 9 in t/dkim2.t at line 50 fail #3
        Checking DKIM_SIGNED
not ok 7
        Checking DKIM_VALID
not ok 8
not ok 9
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/l
ocalrules.tmp -p log/test_default.cf  -t < data/dkim/test-pass-15.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 10 in t/SATest.pm at line 707 fail #7
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 11 in t/SATest.pm at line 707 fail #8
Output can be examined in: log/d.dkim/10
# Failed test 12 in t/dkim2.t at line 50 fail #4
        Checking DKIM_SIGNED
not ok 10
        Checking DKIM_VALID
not ok 11
not ok 12
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/l
ocalrules.tmp -p log/test_default.cf  -t < data/dkim/test-pass-14.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 13 in t/SATest.pm at line 707 fail #9
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 14 in t/SATest.pm at line 707 fail #10
Output can be examined in: log/d.dkim/13
# Failed test 15 in t/dkim2.t at line 50 fail #5
        Checking DKIM_SIGNED
not ok 13
        Checking DKIM_VALID
not ok 14
not ok 15
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-09.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 16 in t/SATest.pm at line 707 fail #11
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 17 in t/SATest.pm at line 707 fail #12
Output can be examined in: log/d.dkim/16
# Failed test 18 in t/dkim2.t at line 50 fail #6
        Checking DKIM_SIGNED
not ok 16
        Checking DKIM_VALID
not ok 17
not ok 18
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-12.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 19 in t/SATest.pm at line 707 fail #13
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 20 in t/SATest.pm at line 707 fail #14
Output can be examined in: log/d.dkim/19
# Failed test 21 in t/dkim2.t at line 50 fail #7
        Checking DKIM_SIGNED
not ok 19
        Checking DKIM_VALID
not ok 20
not ok 21
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-19.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 22 in t/SATest.pm at line 707 fail #15
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 23 in t/SATest.pm at line 707 fail #16
Output can be examined in: log/d.dkim/22
# Failed test 24 in t/dkim2.t at line 50 fail #8
        Checking DKIM_SIGNED
not ok 22
        Checking DKIM_VALID
not ok 23
not ok 24
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-13.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 25 in t/SATest.pm at line 707 fail #17
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 26 in t/SATest.pm at line 707 fail #18
Output can be examined in: log/d.dkim/25
# Failed test 27 in t/dkim2.t at line 50 fail #9
        Checking DKIM_SIGNED
not ok 25
        Checking DKIM_VALID
not ok 26
not ok 27
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-18.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 28 in t/SATest.pm at line 707 fail #19
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 29 in t/SATest.pm at line 707 fail #20
Output can be examined in: log/d.dkim/28
# Failed test 30 in t/dkim2.t at line 50 fail #10
        Checking DKIM_SIGNED
not ok 28
        Checking DKIM_VALID
not ok 29
not ok 30
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-02.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 31 in t/SATest.pm at line 707 fail #21
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 32 in t/SATest.pm at line 707 fail #22
Output can be examined in: log/d.dkim/31
# Failed test 33 in t/dkim2.t at line 50 fail #11
        Checking DKIM_SIGNED
not ok 31
        Checking DKIM_VALID
not ok 32
not ok 33
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-17.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 34 in t/SATest.pm at line 707 fail #23
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 35 in t/SATest.pm at line 707 fail #24
Output can be examined in: log/d.dkim/34
# Failed test 36 in t/dkim2.t at line 50 fail #12
        Checking DKIM_SIGNED
not ok 34
        Checking DKIM_VALID
not ok 35
not ok 36
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-01.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 37 in t/SATest.pm at line 707 fail #25
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 38 in t/SATest.pm at line 707 fail #26
Output can be examined in: log/d.dkim/37
# Failed test 39 in t/dkim2.t at line 50 fail #13
        Checking DKIM_SIGNED
not ok 37
        Checking DKIM_VALID
not ok 38
not ok 39
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-06.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 40 in t/SATest.pm at line 707 fail #27
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 41 in t/SATest.pm at line 707 fail #28
Output can be examined in: log/d.dkim/40
# Failed test 42 in t/dkim2.t at line 50 fail #14
        Checking DKIM_SIGNED
not ok 40
        Checking DKIM_VALID
not ok 41
not ok 42
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-03.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 43 in t/SATest.pm at line 707 fail #29
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 44 in t/SATest.pm at line 707 fail #30
Output can be examined in: log/d.dkim/43
# Failed test 45 in t/dkim2.t at line 50 fail #15
        Checking DKIM_SIGNED
not ok 43
        Checking DKIM_VALID
not ok 44
not ok 45
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-05.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 46 in t/SATest.pm at line 707 fail #31
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 47 in t/SATest.pm at line 707 fail #32
Output can be examined in: log/d.dkim/46
# Failed test 48 in t/dkim2.t at line 50 fail #16
        Checking DKIM_SIGNED
not ok 46
        Checking DKIM_VALID
not ok 47
not ok 48
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-07.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 49 in t/SATest.pm at line 707 fail #33
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 50 in t/SATest.pm at line 707 fail #34
Output can be examined in: log/d.dkim/49
# Failed test 51 in t/dkim2.t at line 50 fail #17
        Checking DKIM_SIGNED
not ok 49
        Checking DKIM_VALID
not ok 50
not ok 51
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-10.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 52 in t/SATest.pm at line 707 fail #35
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 53 in t/SATest.pm at line 707 fail #36
Output can be examined in: log/d.dkim/52
# Failed test 54 in t/dkim2.t at line 50 fail #18
        Checking DKIM_SIGNED
not ok 52
        Checking DKIM_VALID
not ok 53
not ok 54
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-pass-04.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 50.
# Failed test 55 in t/SATest.pm at line 707 fail #37
        Not found: DKIM_VALID =  DKIM_VALID  at t/dkim2.t line 50.
# Failed test 56 in t/SATest.pm at line 707 fail #38
Output can be examined in: log/d.dkim/55
# Failed test 57 in t/dkim2.t at line 50 fail #19
        Checking DKIM_SIGNED
not ok 55
        Checking DKIM_VALID
not ok 56
not ok 57
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-01.msg
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 60.
ok 58
ok 59
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-04.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 72.
# Failed test 60 in t/SATest.pm at line 707 fail #39
Output can be examined in: log/d.dkim/60
# Failed test 62 in t/dkim2.t at line 72
#  t/dkim2.t line 72 is:   ok ok_all_patterns();
        Checking DKIM_SIGNED
not ok 60
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 72.
ok 61
not ok 62
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-08.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 72.
# Failed test 63 in t/SATest.pm at line 707 fail #40
Output can be examined in: log/d.dkim/63
# Failed test 65 in t/dkim2.t at line 72 fail #2
        Checking DKIM_SIGNED
not ok 63
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 72.
ok 64
not ok 65
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-05.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 72.
# Failed test 66 in t/SATest.pm at line 707 fail #41
Output can be examined in: log/d.dkim/66
# Failed test 68 in t/dkim2.t at line 72 fail #3
        Checking DKIM_SIGNED
not ok 66
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 72.
ok 67
not ok 68
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-06.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 72.
# Failed test 69 in t/SATest.pm at line 707 fail #42
Output can be examined in: log/d.dkim/69
# Failed test 71 in t/dkim2.t at line 72 fail #4
        Checking DKIM_SIGNED
not ok 69
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 72.
ok 70
not ok 71
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-02.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 72.
# Failed test 72 in t/SATest.pm at line 707 fail #43
Output can be examined in: log/d.dkim/72
# Failed test 74 in t/dkim2.t at line 72 fail #5
        Checking DKIM_SIGNED
not ok 72
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 72.
ok 73
not ok 74
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-07.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 72.
# Failed test 75 in t/SATest.pm at line 707 fail #44
Output can be examined in: log/d.dkim/75
# Failed test 77 in t/dkim2.t at line 72 fail #6
        Checking DKIM_SIGNED
not ok 75
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 72.
ok 76
not ok 77
        /usr/bin/perl -T -w ../spamassassin.raw -C log/test_rules_copy
 --siteconfigpath log/localrules.tmp -p log/test_default.cf  -t <
data/dkim/test-fail-03.msg
        Not found: DKIM_SIGNED =  DKIM_SIGNED  at t/dkim2.t line 72.
# Failed test 78 in t/SATest.pm at line 707 fail #45
Output can be examined in: log/d.dkim/78
# Failed test 80 in t/dkim2.t at line 72 fail #7
        Checking DKIM_SIGNED
not ok 78
        Checking for anti-pattern DKIM_VALID at t/dkim2.t line 72.
ok 79
not ok 80
FAILED tests 1-57, 60, 62-63, 65-66, 68-69, 71-72, 74-75, 77-78, 80
        Failed 71/80 tests, 11.25% okay
Failed Test Stat Wstat Total Fail  Failed  List of Failed
-------------------------------------------------------------------------------
t/dkim2.t                 80   71  88.75%  1-57 60 62-63 65-66 68-69 71-72 74-
                                           75 77-78 80
Failed 1/1 test scripts, 0.00% okay. 71/80 subtests failed, 11.25% okay.

> Please do try with Mail::DKIM 0.36, to see if that helps.
>
> Btw, you could run the following one-liner, giving it on stdin
> each of the 'pass' sample messages int/data/dkim/ :
>
>
> perl -MMail::DKIM::Verifier -ne '
>  BEGIN{$dkim=Mail::DKIM::Verifier->new_object};
>  s/\r?\n\z/\015\012/; $dkim->PRINT($_); END{$dkim->CLOSE;
>  printf("%s\n",$_->result_detail) for $dkim->signatures}'
>   < t/data/dkim/test-pass-19.msg

Can't locate object method "signatures" via package
"Mail::DKIM::Verifier" at -e line 1, <> line 14.
END failed--call queue aborted, <> line 14.

--j.

[Mark Martinec]

On Thu, 02 Jul 2009 13:09:46 +0200, Justin Mason <jm@...> wrote:

> 2009/7/2 Mark Martinec <Mark.Martinec+sa@...>:
>> On Thu, 02 Jul 2009 00:13:52 +0200, Kevin A. McGrail <kmcgrail@...>
>> wrote:
>>
>>> I'm running long tests and net tests and getting a lot of failures in the
>>> DKIM area.  I'm running 0.28.
>>>
>>> Is this a known issue?  Should I upgrade DKIM and try again which would
>>> like make me suggest a higher version requirement for the makefile.
>>
>> Don't know, it passes all tests for me.
>> Does it happen to others too?
>
> yep, me too.

There are no DKIM-related rules in the testing .cf file.

Something like the following should do, although I'm not sure
what is the right place to put these:

at the end of data/01_test_rules.cf place the:


ifplugin Mail::SpamAssassin::Plugin::DKIM
full DKIM_SIGNED        eval:check_dkim_signed()
full DKIM_VALID         eval:check_dkim_valid()
full DKIM_VALID_AU      eval:check_dkim_valid_author_sig()
endif


   Mark

[Mark Martinec]

> r790749 | jm | 2009-07-03 00:19:05 +0200 (Fri, 03 Jul 2009) | 1 line
> improve the dkim2.t test situation;
> I now get only tests 11-12 failing

Thanks for updating the data/01_test_rules.cf.

> I now get only tests 11-12 failing

I see that the order of tests depended on a file system
(directory order), which isn't very useful. Fixed:

  Speed up the t/dkim2.t by 40% by using a persistent SA object and
  avoid starting a command line spamassassin for each test message.
  Do tests in sorted order of test messages instead of in the
  seemingly random order of files as read from a directory.
  Print out a name of each test message.
Sending        t/dkim2.t
Committed revision 791499.

I'd expect the failing message to be the test-pass-07.msg
if you were using an old version of Mail::DKIM module.

(btw, as I'm not very familiar with conventions used in
tests, it may be useful if somebody glances over the
updated dkim2.t, especially its BEGIN section, as I just
copied it from somewhere else, not knowing wheather it is
really needed in this test)

  Mark