rt2tort3 import errors

[Dominic Hargreaves]

Hi,

I'm working through the migration of our relatively large RT2 data
set (500,000 tickets) with rt2tort3.

I've already fixed a number of issues with the scripts and I plan to
submit anything relevant as patches once I've had a chance to get
through it all, but there are a couple of things I've hit that I
don't understand.

I've managed a complete export and import cycle okay, and I'm now
testing the incremental mode (which will be required to get the
migration done without excessive downtime). Most tickets are getting
apparently imported succesfully but this one barfs:

Previously this was prefixed with an error (that may have been
incidentally exposed) regarding the now-defunct TicketCustomFieldValues
table. I've now fixed the import script to use use the
ObjectCustomFieldValues instead, and just get:

(queue name and subject name elided)

t-1395038: w[Tue Nov  3 15:36:51 2009] [crit]: Couldn't set EffectiveId: That is already the current value (/usr/share/request-tracker3.8/lib/RT/Ticket_Overlay.pm:500)

Couldn't create TICKET: Ticket could not be created due to an internal error$VAR1 = {
          'Status' => 'resolved',
          'Queue' => 'xxxx',
          'Started' => '2009-10-12 15:12:59+00',
          'Starts' => '1970-01-01 00:00:00+00',
          '_RecordTransaction' => '0',
          'id' => '1395038',
          'LastUpdated' => '2009-10-12 15:13:00+00',
          'Requestor' => [
                         '132522'
                       ],
          'Subject' => 'xxxx',
          'Creator' => undef,
          'Owner' => undef,
          'LastUpdatedBy' => undef,
          'EffectiveId' => '1395038',
          'Resolved' => '2009-10-12 15:12:59+00',
          'Created' => '2009-09-27 18:07:14+00',
          'Due' => '2009-09-27 18:07:14+00'
        };
[Tue Nov  3 15:36:51 2009] [crit]: Died at ./dumpfile-to-rt-3.0 line 700. (/usr/share/request-tracker3.8/lib/RT.pm:387)
Died at ./dumpfile-to-rt-3.0 line 700.

I'm completely failing to make any sense of the error here:

Couldn't set EffectiveId: That is already the current value

since this ticket, which has Id == EffectiveId, is like most other
tickets in this regard, so I can't see even see what's unusual about
this ticket to trigger the bug.

The second problem that I've not completely sorted out is that
most of the ticket imports spew:

[Tue Nov  3 14:40:08 2009] [warning]: Use of uninitialized value $MIMEType in pattern match (m//) at /usr/share/request-tracker3.8/lib/RT/Record.pm line 736. (/usr/share/request-tracker3.8/lib/RT/Record.pm:736)
[Tue Nov  3 14:40:08 2009] [warning]: Use of uninitialized value $Body in length at /usr/share/request-tracker3.8/lib/RT/Record.pm line 753. (/usr/share/request-tracker3.8/lib/RT/Record.pm:753)
[Tue Nov  3 14:40:08 2009] [warning]: Use of uninitialized value in subroutine entry at /usr/share/request-tracker3.8/lib/RT/Record.pm line 783. (/usr/share/request-tracker3.8/lib/RT/Record.pm:783)

I've already squelched a couple of further uninitialized value warnings
in dumpfile-to-rt3.0 itself (on $a->{'ContentEncoding'}) but I'm not
sure of the *correct* way to fix these cases (I could just default it
to an empty string but that doesn't seem right, really).

Any insights or hints would be most appreciated.

Versions, etc:

Live RT2 install: Debian sarge, postgres 7.4
RT2 migrate install: Debian lenny, postgres 7.4 (with suitably old
DBIx::SearchBuilder etc)
RT3 test install: Debian lenny, postgres 8.3, RT 3.8.5 (haven't moved
to 3.8.6 yet as I wanted to complete one round of testing beforehand).

Thanks,
Dominic.

--
Dominic Hargreaves, Systems Development and Support Team
Computing Services, University of Oxford
_______________________________________________
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@...


Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

[Jesse Vincent]

> I'm completely failing to make any sense of the error here:
>
> Couldn't set EffectiveId: That is already the current value

Next step, I think, is to get RT to log stack traces for errors:

http://wiki.bestpractical.com/view/LogsConfig has instructions.
With that, we may have a better idea of what's going wrong.

-j
_______________________________________________
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@...


Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

[Dominic Hargreaves]

On Tue, Nov 03, 2009 at 11:30:17AM -0500, Jesse Vincent wrote:
> > I'm completely failing to make any sense of the error here:
> >
> > Couldn't set EffectiveId: That is already the current value
>
> Next step, I think, is to get RT to log stack traces for errors:
>
> http://wiki.bestpractical.com/view/LogsConfig has instructions.
> With that, we may have a better idea of what's going wrong.

*nod*

Here's the same error again with stack traces turned on:

t-1395038: w[Tue Nov  3 17:19:23 2009] [crit]: Couldn't set EffectiveId: That is already the current value (/usr/share/request-tracker3.8/lib/RT/Ticket_Overlay.pm:500)
Trace begun at /usr/share/request-tracker3.8/lib/RT.pm line 300
Log::Dispatch::__ANON__('Log::Dispatch=HASH(0x34207e0)', 'Couldn\'t set EffectiveId: That is already the current value') called at /usr/share/request-tracker3.8/lib/RT/Ticket_Overlay.pm line 500
RT::Ticket::Create('RT::Ticket=HASH(0x7511ba8)', 'Status', 'resolved', 'Queue', 'xxxx', 'Started', '2009-10-12 15:12:59+00', 'Starts', '1970-01-01 00:00:00+00', '_RecordTransaction', 0, 'id', 1395038, 'LastUpdated', '2009-10-12 15:13:00+00', 'Requestor', 'ARRAY(0x80c5e88)', 'Subject', 'xxxx', 'Creator', undef, 'Owner', undef, 'LastUpdatedBy', undef, 'EffectiveId', 1395038, 'Resolved', '2009-10-12 15:12:59+00', 'Created', '2009-09-27 18:07:14+00', 'Due', '2009-09-27 18:07:14+00') called at dumpfile-to-rt-3.0 line 695
RT::import_tickets at dumpfile-to-rt-3.0 line 89

Couldn't create TICKET: Ticket could not be created due to an internal error$VAR1 = {
          'Status' => 'resolved',
          'Queue' => 'xxxx',
          'Started' => '2009-10-12 15:12:59+00',
          'Starts' => '1970-01-01 00:00:00+00',
          '_RecordTransaction' => '0',
          'id' => '1395038',
          'LastUpdated' => '2009-10-12 15:13:00+00',
          'Requestor' => [
                         '132522'
                       ],
          'Subject' => 'xxxx',
          'Creator' => undef,
          'Owner' => undef,
          'LastUpdatedBy' => undef,
          'EffectiveId' => '1395038',
          'Resolved' => '2009-10-12 15:12:59+00',
          'Created' => '2009-09-27 18:07:14+00',
          'Due' => '2009-09-27 18:07:14+00'
        };
[Tue Nov  3 17:19:23 2009] [crit]: Died at ./dumpfile-to-rt-3.0 line 700. (/usr/share/request-tracker3.8/lib/RT.pm:387)
Trace begun at /usr/share/request-tracker3.8/lib/RT.pm line 300
Log::Dispatch::__ANON__('Log::Dispatch=HASH(0x34207e0)', 'Died at ./dumpfile-to-rt-3.0 line 700.^J') called at /usr/share/request-tracker3.8/lib/RT.pm line 387
RT::__ANON__('Died at ./dumpfile-to-rt-3.0 line 700.^J') called at dumpfile-to-rt-3.0 line 700
RT::import_tickets at dumpfile-to-rt-3.0 line 89
Died at ./dumpfile-to-rt-3.0 line 700.

If I try to insert the same ticket manually, things seem to proceed
without incident:

#!/usr/bin/perl

use strict;
use warnings;

use lib ("/usr/share/request-tracker3.8/lib");

use RT::Interface::CLI qw(CleanEnv GetCurrentUser GetMessageContent loc);
use RT::Tickets;
use RT::Template;
use RT::CustomFields;
use RT::Principals;
use RT::Scrips;

CleanEnv();
RT::LoadConfig();
RT::Init();

$RT::LogStackTraces = 1;

my $tick_object = RT::Ticket->new($RT::SystemUser);

my $ticket = {
          'Status' => 'resolved',
          'Queue' => 'xxxx',
          'Started' => '2009-10-12 15:12:59+00',
          'Starts' => '1970-01-01 00:00:00+00',
          '_RecordTransaction' => '0',
          'id' => '1395038',
          'LastUpdated' => '2009-10-12 15:13:00+00',
          'Requestor' => [
                         '132522'
                       ],
          'Subject' => 'xxxx',
          'Creator' => undef,
          'Owner' => undef,
          'LastUpdatedBy' => undef,
          'EffectiveId' => '1395038',
          'Resolved' => '2009-10-12 15:12:59+00',
          'Created' => '2009-09-27 18:07:14+00',
          'Due' => '2009-09-27 18:07:14+00'
        };

$tick_object->Create( %{$ticket} );

rt-migrate@steely-glint:~$ ~dom/test-create
Name "RT::LogStackTraces" used only once: possible typo at /home/dom/test-create line 19.
rt-migrate@steely-glint:~$

(and the ticket is created in the database).

the error comes from the second argument returned by $self->__Set
in Ticket_Overlay.pm, which is in turn from DBIx::SearchBuilder::Record:

I simply cannot see how that error could occur when creating a new
ticket in Ticket_Overlay.pm, reading through the logic.
Ticket.pm creates the ticket defaulting EffectiveId to 0, and
Ticket_Overlay.pm then tries to set it to the EffectiveId to that
is given in the arguments to Create, which is not 0.

The plot thickens (and it's time for me to go home :)

--
Dominic Hargreaves, Systems Development and Support Team
Computing Services, University of Oxford
_______________________________________________
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@...


Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

[Jesse Vincent]

On Tue, Nov 03, 2009 at 06:12:58PM +0000, Dominic Hargreaves wrote:

> On Tue, Nov 03, 2009 at 11:30:17AM -0500, Jesse Vincent wrote:
> > > I'm completely failing to make any sense of the error here:
> > >
> > > Couldn't set EffectiveId: That is already the current value
> >
> > Next step, I think, is to get RT to log stack traces for errors:
> >
> > http://wiki.bestpractical.com/view/LogsConfig has instructions.
> > With that, we may have a better idea of what's going wrong.
>
> *nod*
>
> Here's the same error again with stack traces turned on:

This is, indeed, a bit weird.  The next step is likely instrumenting
around line 500 of Ticket_Overlay.pm

-j
_______________________________________________
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@...


Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

[Dominic Hargreaves]

On Wed, Nov 04, 2009 at 02:20:17PM -0500, Jesse Vincent wrote:

> This is, indeed, a bit weird.  The next step is likely instrumenting
> around line 500 of Ticket_Overlay.pm

Turns out that this problem was a caching error. rt2tort3 had
not been updated to take account of

http://github.com/bestpractical/rt/commit/e252f7ff5202b710dc3339dba2f950df802540e8

(ironic, since

http://github.com/bestpractical/rt/commit/3f5b52e511574fb2d350b94957c8b1566fcb473b

suggests that that config was put in specifically for the importer :)

I thought that I had read that people had used rt2tort3 successfully
with RT3.8 but maybe I'm wrong. Or possibly noone has used the
incremental mode?

I've a number of patches to rt2tort3 I'd like to feed upstream;
what's the best place for them? rt.cpan.org?

--
Dominic Hargreaves, Systems Development and Support Team
Computing Services, University of Oxford
_______________________________________________
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@...


Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

[Jesse Vincent]

>
> I've a number of patches to rt2tort3 I'd like to feed upstream;

Fantastic.

> what's the best place for them? rt.cpan.org?

Nominally yes. But it depends a bit on what you're using for
development. If it happens to be a public git repo, I can just pull from
you.

-j

_______________________________________________
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@...


Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
Buy a copy at http://rtbook.bestpractical.com